2024-04-25T01:14:39.361 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T01:14:39.361 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/auto-repair} --verbose --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319 --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319/orig.config.yaml 2024-04-25T01:14:39.407 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:14:39.496 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319 branch: squid description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/auto-repair} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672319' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - force file system read-only - bad backtrace - MDS in read-only mode - \(MDS_READ_ONLY\) sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF0BLrIbcXB1/3PN9Zx1XwhG5TqPqQIE/fkSgDgLQuYq8F3wbH5vKKhUlRH3Yz+h5Rwrrwflj7TOIQfGgDW5bYs= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAexl1tvEKG02Tx7lY8UrCME3hEOnCIZCTBvU7BK2Md/vEyXCf3aMWUEqWanYfdYNqi8BRdrnacvNTb4cqEiqbc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_auto_repair teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T01:14:39.497 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-25T01:14:39.497 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-25T01:14:39.498 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T01:14:39.500 INFO:teuthology.task.internal:Checking packages... 2024-04-25T01:14:39.527 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-25T01:14:39.527 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T01:14:39.527 INFO:teuthology.packaging:ref: None 2024-04-25T01:14:39.527 INFO:teuthology.packaging:tag: None 2024-04-25T01:14:39.528 INFO:teuthology.packaging:branch: squid 2024-04-25T01:14:39.528 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:14:39.528 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-25T01:14:39.843 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-25T01:14:39.845 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T01:14:39.851 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T01:14:39.851 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T01:14:39.857 INFO:teuthology.task.internal:Saving configuration 2024-04-25T01:14:39.878 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T01:14:39.885 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T01:14:39.911 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:09:45.810026', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF0BLrIbcXB1/3PN9Zx1XwhG5TqPqQIE/fkSgDgLQuYq8F3wbH5vKKhUlRH3Yz+h5Rwrrwflj7TOIQfGgDW5bYs='} 2024-04-25T01:14:39.934 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:09:45.808751', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAexl1tvEKG02Tx7lY8UrCME3hEOnCIZCTBvU7BK2Md/vEyXCf3aMWUEqWanYfdYNqi8BRdrnacvNTb4cqEiqbc='} 2024-04-25T01:14:39.934 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T01:14:39.940 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-25T01:14:39.940 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-25T01:14:39.940 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T01:14:40.016 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff9b89550d0>, signals=[15]) 2024-04-25T01:14:40.016 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T01:14:40.022 INFO:teuthology.task.internal:Opening connections... 2024-04-25T01:14:40.022 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:14:40.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:14:40.111 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:14:40.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:14:40.186 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T01:14:40.192 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-04-25T01:14:40.210 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-04-25T01:14:40.210 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-04-25T01:14:40.268 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-04-25T01:14:40.269 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="9" 2024-04-25T01:14:40.269 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-04-25T01:14:40.269 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:14:40.269 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="9" 2024-04-25T01:14:40.269 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:14:40.269 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:14:40.269 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-04-25T01:14:40.270 INFO:teuthology.orchestra.run.smithi077.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:14:40.270 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:14:40.270 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:14:40.270 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:14:40.270 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:14:40.270 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:14:40.272 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-04-25T01:14:40.365 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-04-25T01:14:40.384 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-04-25T01:14:40.384 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-04-25T01:14:40.442 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-04-25T01:14:40.442 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-04-25T01:14:40.442 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-04-25T01:14:40.442 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:14:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-04-25T01:14:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:14:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:14:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-04-25T01:14:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:14:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:14:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:14:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:14:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:14:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:14:40.444 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-04-25T01:14:40.468 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T01:14:40.478 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T01:14:40.483 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T01:14:40.484 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:14:40.486 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:14:40.502 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T01:14:40.508 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T01:14:40.508 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:14:40.543 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:14:40.570 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T01:14:40.632 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T01:14:40.647 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-25T01:14:40.648 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-04-25T01:14:40.648 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-04-25T01:14:40.648 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:14:40.649 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:14:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:14:40.664 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-04-25T01:14:40.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:14:40.666 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-04-25T01:14:40.721 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:14:40.721 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-437.el9.x86_64 2024-04-25T01:14:40.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-04-25T01:14:40.723 INFO:teuthology.orchestra.run.smithi077.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:14:40.723 INFO:teuthology.task.kernel:Running kernel on smithi077: 5.14.0-437.el9.x86_64 2024-04-25T01:14:40.723 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2024-04-25T01:14:43.029 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-25T01:14:43.111 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-25T01:14:45.470 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-25T01:14:49.637 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T01:14:51.154 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 16 kB 00:00 2024-04-25T01:14:52.114 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-04-25T01:14:57.123 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T01:14:59.082 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-25T01:14:59.084 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-25T01:14:59.084 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-25T01:14:59.193 DEBUG:teuthology.orchestra.run.smithi121:> echo no | sudo yum reinstall kernel || true 2024-04-25T01:14:59.727 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:02 ago on Thu 25 Apr 2024 01:14:57 AM UTC. 2024-04-25T01:14:59.845 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-25T01:14:59.857 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-25T01:14:59.857 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-25T01:14:59.857 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-25T01:14:59.857 INFO:teuthology.orchestra.run.smithi121.stdout:Reinstalling: 2024-04-25T01:14:59.857 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:14:59.857 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:14:59.857 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-25T01:14:59.857 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-25T01:14:59.858 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:14:59.858 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 255 k 2024-04-25T01:14:59.858 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 0 2024-04-25T01:14:59.858 INFO:teuthology.orchestra.run.smithi121.stderr:Operation aborted. 2024-04-25T01:14:59.909 INFO:teuthology.orchestra.run.smithi121.stdout:Is this ok [y/N]: 2024-04-25T01:14:59.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum reinstall -y kernel || true 2024-04-25T01:14:59.920 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-04-25T01:15:00.402 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:14:57 AM UTC. 2024-04-25T01:15:00.518 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-25T01:15:00.518 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-25T01:15:00.519 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-25T01:15:00.519 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-25T01:15:00.519 INFO:teuthology.orchestra.run.smithi121.stdout:Reinstalling: 2024-04-25T01:15:00.519 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:15:00.519 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:15:00.519 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-25T01:15:00.519 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-25T01:15:00.519 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:15:00.520 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 255 k 2024-04-25T01:15:00.520 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 0 2024-04-25T01:15:00.520 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-25T01:15:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.6 MB/s | 255 kB 00:00 2024-04-25T01:15:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:15:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:Total 941 kB/s | 255 kB 00:00 2024-04-25T01:15:00.807 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-25T01:15:01.000 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-25T01:15:01.000 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-25T01:15:01.005 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-25T01:15:01.006 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-25T01:15:01.145 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-25T01:15:01.216 INFO:teuthology.orchestra.run.smithi121.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:15:01.216 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:15:01.875 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:15:01.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:15:02.545 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:15:02.546 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:15:02.546 INFO:teuthology.orchestra.run.smithi121.stdout:Reinstalled: 2024-04-25T01:15:02.546 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T01:15:02.546 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:15:02.547 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-25T01:15:02.710 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T01:15:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T01:15:02.757 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T01:15:02.757 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T01:15:02.757 DEBUG:teuthology.parallel:result is None 2024-04-25T01:15:04.529 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 6.0 MB/s | 6.2 MB 00:01 2024-04-25T01:15:06.388 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-25T01:15:08.065 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-25T01:15:13.123 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-25T01:15:14.996 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T01:15:14.996 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T01:15:15.086 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-04-25T01:15:15.087 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-04-25T01:15:15.087 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-04-25T01:15:15.205 DEBUG:teuthology.orchestra.run.smithi077:> echo no | sudo yum reinstall kernel || true 2024-04-25T01:15:15.702 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:02 ago on Thu 25 Apr 2024 01:15:13 AM UTC. 2024-04-25T01:15:15.824 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-04-25T01:15:15.840 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-25T01:15:15.840 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-04-25T01:15:15.840 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-25T01:15:15.840 INFO:teuthology.orchestra.run.smithi077.stdout:Reinstalling: 2024-04-25T01:15:15.840 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:15:15.840 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:15:15.840 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-04-25T01:15:15.840 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-25T01:15:15.840 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:15:15.840 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 255 k 2024-04-25T01:15:15.840 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 0 2024-04-25T01:15:15.841 INFO:teuthology.orchestra.run.smithi077.stderr:Operation aborted. 2024-04-25T01:15:15.892 INFO:teuthology.orchestra.run.smithi077.stdout:Is this ok [y/N]: 2024-04-25T01:15:15.893 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum reinstall -y kernel || true 2024-04-25T01:15:16.385 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:15:13 AM UTC. 2024-04-25T01:15:16.499 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-04-25T01:15:16.500 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-25T01:15:16.500 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-04-25T01:15:16.500 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-25T01:15:16.500 INFO:teuthology.orchestra.run.smithi077.stdout:Reinstalling: 2024-04-25T01:15:16.500 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:15:16.500 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:15:16.501 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-04-25T01:15:16.501 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-25T01:15:16.501 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:15:16.501 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 255 k 2024-04-25T01:15:16.501 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 0 2024-04-25T01:15:16.501 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-04-25T01:15:16.784 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.9 MB/s | 255 kB 00:00 2024-04-25T01:15:16.784 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:15:16.784 INFO:teuthology.orchestra.run.smithi077.stdout:Total 900 kB/s | 255 kB 00:00 2024-04-25T01:15:16.800 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-04-25T01:15:16.973 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-04-25T01:15:16.973 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-04-25T01:15:16.978 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-04-25T01:15:16.978 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-04-25T01:15:17.115 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-04-25T01:15:17.185 INFO:teuthology.orchestra.run.smithi077.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:15:17.185 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:15:17.875 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:15:17.875 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:15:18.577 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:15:18.578 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:15:18.578 INFO:teuthology.orchestra.run.smithi077.stdout:Reinstalled: 2024-04-25T01:15:18.578 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T01:15:18.578 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:15:18.578 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-04-25T01:15:18.746 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T01:15:18.791 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T01:15:18.792 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T01:15:18.792 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T01:15:18.792 DEBUG:teuthology.parallel:result is None 2024-04-25T01:15:18.792 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T01:15:18.803 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T01:15:18.804 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:15:18.806 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:15:18.824 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T01:15:18.836 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T01:15:18.849 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T01:15:18.849 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:15:18.861 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:15:18.981 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T01:15:18.993 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T01:15:18.993 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:15:18.996 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:15:19.025 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:15:19.033 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:15:19.043 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:15:19.047 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:15:19.049 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T01:15:19.062 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T01:15:19.063 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:15:19.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:15:19.115 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T01:15:19.132 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T01:15:19.132 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:15:19.152 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:15:19.171 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:15:19.250 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:15:19.304 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:15:19.388 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:15:19.425 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:15:19.425 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:15:19.487 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:15:19.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:15:19.582 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:15:19.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:15:19.709 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:15:19.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:15:19.775 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-04-25T01:15:19.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-04-25T01:15:19.812 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:15:19.857 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:15:20.367 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T01:15:20.382 INFO:teuthology.task.internal:Starting timer... 2024-04-25T01:15:20.382 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T01:15:20.410 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T01:15:20.432 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T01:15:20.432 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-04-25T01:15:20.503 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-04-25T01:15:20.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-04-25T01:15:20.576 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-04-25T01:15:20.578 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T01:15:20.579 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-04-25T01:15:20.603 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-04-25T01:15:20.604 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-04-25T01:15:20.630 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-04-25T01:15:20.630 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive'} 2024-04-25T01:15:20.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:15:20.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:15:20.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:15:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:15:20.697 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T01:15:20.697 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:15:20.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:15:20.762 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T01:15:21.073 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T01:15:21.082 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T01:15:21.245 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T01:15:21.266 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T01:15:21.267 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T01:18:50.696 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2024-04-25T01:18:50.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-25T01:18:50.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:18:50.796 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-04-25T01:18:50.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-25T01:18:50.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-25T01:18:50.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:18:50.951 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-04-25T01:18:51.028 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-25T01:18:51.028 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T01:18:51.040 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T01:18:51.040 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:18:51.041 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:18:51.043 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:18:51.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:18:51.069 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:18:51.082 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:18:51.095 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:18:51.104 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-04-25T01:18:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:18:51.114 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-04-25T01:18:51.127 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:18:51.132 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-04-25T01:18:51.140 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:18:51.143 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-04-25T01:18:51.157 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:18:51.171 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:18:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:18:51.197 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:18:51.197 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-04-25T01:18:51.197 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:51.197 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:51.197 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:51.197 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:51.236 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:18:51.240 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:18:51.240 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-04-25T01:18:51.240 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:51.240 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:51.240 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:51.240 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:51.241 INFO:teuthology.run_tasks:Running task install... 2024-04-25T01:18:51.250 DEBUG:teuthology.task.install:project ceph 2024-04-25T01:18:51.251 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-25T01:18:51.251 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-25T01:18:51.251 INFO:teuthology.task.install:Using flavor: default 2024-04-25T01:18:51.258 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T01:18:51.258 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T01:18:51.259 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:18:51.260 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:18:51.262 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:18:51.263 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:18:51.490 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:18:51.490 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:18:51.495 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:18:51.496 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:18:51.611 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:18:51.612 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:18:51.612 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:18:51.636 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:18:51.637 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:18:51.662 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:18:51.662 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:18:51.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:18:51.687 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:18:51.688 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:18:51.701 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:18:51.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:18:51.773 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:18:51.824 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:18:51.837 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-04-25T01:18:51.838 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-04-25T01:18:51.887 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-04-25T01:18:51.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-04-25T01:18:52.063 INFO:teuthology.orchestra.run.smithi077.stdout:48 files removed 2024-04-25T01:18:52.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:18:52.113 INFO:teuthology.orchestra.run.smithi121.stdout:48 files removed 2024-04-25T01:18:52.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:18:52.715 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 262 kB/s | 83 kB 00:00 2024-04-25T01:18:52.768 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 197 kB/s | 83 kB 00:00 2024-04-25T01:18:53.140 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-04-25T01:18:53.243 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 27 kB/s | 11 kB 00:00 2024-04-25T01:18:53.608 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 4.4 kB/s | 1.9 kB 00:00 2024-04-25T01:18:53.628 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-04-25T01:18:54.333 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-25T01:18:55.336 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.0 MB 00:01 2024-04-25T01:18:57.013 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-04-25T01:18:58.975 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 19 MB 00:03 2024-04-25T01:19:02.422 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-04-25T01:19:03.509 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 6.2 MB 00:00 2024-04-25T01:19:03.881 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-04-25T01:19:05.826 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-04-25T01:19:05.866 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-25T01:19:07.077 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-25T01:19:10.907 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-25T01:19:12.219 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-25T01:19:13.034 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:19:13.035 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:19:13.046 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:19:13.046 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:19:13.047 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:19:13.048 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:19:13.050 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:19:13.052 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:19:13.053 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:19:13.113 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:19:13.125 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:19:13.126 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:19:13.129 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:19:13.130 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-04-25T01:19:13.134 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-04-25T01:19:13.135 INFO:teuthology.orchestra.run.smithi077.stdout:Install 148 Packages 2024-04-25T01:19:13.135 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 4 Packages 2024-04-25T01:19:13.135 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:19:13.135 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 220 M 2024-04-25T01:19:13.135 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-04-25T01:19:14.319 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:19:14.320 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:19:14.330 INFO:teuthology.orchestra.run.smithi121.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:19:14.331 INFO:teuthology.orchestra.run.smithi121.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:19:14.332 INFO:teuthology.orchestra.run.smithi121.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:19:14.333 INFO:teuthology.orchestra.run.smithi121.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:19:14.334 INFO:teuthology.orchestra.run.smithi121.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:19:14.336 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:19:14.337 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:19:14.392 INFO:teuthology.orchestra.run.smithi077.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-04-25T01:19:14.397 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-25T01:19:14.406 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-04-25T01:19:14.406 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-25T01:19:14.406 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-04-25T01:19:14.406 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-25T01:19:14.406 INFO:teuthology.orchestra.run.smithi121.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:19:14.406 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:19:14.406 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:19:14.406 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:19:14.406 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:19:14.407 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:19:14.408 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-25T01:19:14.409 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:19:14.410 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:19:14.411 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:19:14.412 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:19:14.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:19:14.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:19:14.415 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:19:14.416 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:19:14.417 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:19:14.418 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-25T01:19:14.418 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-04-25T01:19:14.418 INFO:teuthology.orchestra.run.smithi121.stdout:Install 148 Packages 2024-04-25T01:19:14.418 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 4 Packages 2024-04-25T01:19:14.418 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:19:14.418 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 220 M 2024-04-25T01:19:14.418 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-25T01:19:15.301 INFO:teuthology.orchestra.run.smithi077.stdout:(2/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 16 MB/s | 21 MB 00:01 2024-04-25T01:19:15.351 INFO:teuthology.orchestra.run.smithi077.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 2.8 MB/s | 145 kB 00:00 2024-04-25T01:19:15.385 INFO:teuthology.orchestra.run.smithi077.stdout:(4/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 897 kB/s | 890 kB 00:00 2024-04-25T01:19:15.502 INFO:teuthology.orchestra.run.smithi077.stdout:(5/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 16 MB/s | 2.4 MB 00:00 2024-04-25T01:19:15.661 INFO:teuthology.orchestra.run.smithi121.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-04-25T01:19:15.760 INFO:teuthology.orchestra.run.smithi077.stdout:(6/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 18 MB/s | 4.7 MB 00:00 2024-04-25T01:19:16.070 INFO:teuthology.orchestra.run.smithi121.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 2.1 MB/s | 890 kB 00:00 2024-04-25T01:19:16.145 INFO:teuthology.orchestra.run.smithi121.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.9 MB/s | 145 kB 00:00 2024-04-25T01:19:16.511 INFO:teuthology.orchestra.run.smithi077.stdout:(7/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 1.3 MB/s | 1.5 MB 00:01 2024-04-25T01:19:16.620 INFO:teuthology.orchestra.run.smithi121.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 5.0 MB/s | 2.4 MB 00:00 2024-04-25T01:19:16.744 INFO:teuthology.orchestra.run.smithi077.stdout:(8/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 17 MB/s | 17 MB 00:00 2024-04-25T01:19:16.762 INFO:teuthology.orchestra.run.smithi121.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 11 MB/s | 1.5 MB 00:00 2024-04-25T01:19:16.786 INFO:teuthology.orchestra.run.smithi077.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 609 kB/s | 25 kB 00:00 2024-04-25T01:19:17.646 INFO:teuthology.orchestra.run.smithi121.stdout:(6/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 5.3 MB/s | 4.7 MB 00:00 2024-04-25T01:19:18.253 INFO:teuthology.orchestra.run.smithi077.stdout:(10/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x 1.3 MB/s | 5.4 MB 00:04 2024-04-25T01:19:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 889 kB/s | 222 kB 00:00 2024-04-25T01:19:18.661 INFO:teuthology.orchestra.run.smithi077.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 171 kB/s | 27 kB 00:00 2024-04-25T01:19:18.795 INFO:teuthology.orchestra.run.smithi077.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 238 kB/s | 32 kB 00:00 2024-04-25T01:19:19.362 INFO:teuthology.orchestra.run.smithi121.stdout:(7/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 1.3 MB/s | 5.4 MB 00:04 2024-04-25T01:19:19.770 INFO:teuthology.orchestra.run.smithi077.stdout:(14/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 16 MB/s | 48 MB 00:02 2024-04-25T01:19:19.804 INFO:teuthology.orchestra.run.smithi077.stdout:(15/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 739 kB/s | 745 kB 00:01 2024-04-25T01:19:19.829 INFO:teuthology.orchestra.run.smithi077.stdout:(16/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.7 MB/s | 163 kB 00:00 2024-04-25T01:19:19.888 INFO:teuthology.orchestra.run.smithi077.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 8.4 MB/s | 501 kB 00:00 2024-04-25T01:19:19.904 INFO:teuthology.orchestra.run.smithi121.stdout:(8/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 7.4 MB/s | 17 MB 00:02 2024-04-25T01:19:19.946 INFO:teuthology.orchestra.run.smithi077.stdout:(18/152): librados-devel-19.0.0-2445.gd2c45d97. 891 kB/s | 127 kB 00:00 2024-04-25T01:19:19.996 INFO:teuthology.orchestra.run.smithi121.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 275 kB/s | 25 kB 00:00 2024-04-25T01:19:20.047 INFO:teuthology.orchestra.run.smithi077.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 452 kB/s | 45 kB 00:00 2024-04-25T01:19:20.311 INFO:teuthology.orchestra.run.smithi077.stdout:(20/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 13 MB/s | 5.3 MB 00:00 2024-04-25T01:19:20.381 INFO:teuthology.orchestra.run.smithi077.stdout:(21/152): python3-ceph-common-19.0.0-2445.gd2c4 398 kB/s | 133 kB 00:00 2024-04-25T01:19:20.406 INFO:teuthology.orchestra.run.smithi077.stdout:(22/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.7 MB/s | 162 kB 00:00 2024-04-25T01:19:20.490 INFO:teuthology.orchestra.run.smithi077.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.5 MB/s | 299 kB 00:00 2024-04-25T01:19:20.565 INFO:teuthology.orchestra.run.smithi077.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 99 kB 00:00 2024-04-25T01:19:20.641 INFO:teuthology.orchestra.run.smithi077.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-25T01:19:20.899 INFO:teuthology.orchestra.run.smithi077.stdout:(26/152): python3-rados-19.0.0-2445.gd2c45d97.e 620 kB/s | 321 kB 00:00 2024-04-25T01:19:21.066 INFO:teuthology.orchestra.run.smithi077.stdout:(27/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 7.3 MB/s | 3.1 MB 00:00 2024-04-25T01:19:21.125 INFO:teuthology.orchestra.run.smithi077.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 541 kB/s | 32 kB 00:00 2024-04-25T01:19:21.167 INFO:teuthology.orchestra.run.smithi077.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 3.3 MB/s | 141 kB 00:00 2024-04-25T01:19:21.409 INFO:teuthology.orchestra.run.smithi077.stdout:(30/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 336 kB/s | 171 kB 00:00 2024-04-25T01:19:21.543 INFO:teuthology.orchestra.run.smithi077.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 9.4 MB/s | 3.5 MB 00:00 2024-04-25T01:19:21.618 INFO:teuthology.orchestra.run.smithi077.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 3.2 MB/s | 249 kB 00:00 2024-04-25T01:19:21.693 INFO:teuthology.orchestra.run.smithi077.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 657 kB/s | 49 kB 00:00 2024-04-25T01:19:21.777 INFO:teuthology.orchestra.run.smithi077.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 184 kB/s | 15 kB 00:00 2024-04-25T01:19:21.877 INFO:teuthology.orchestra.run.smithi077.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 2.6 MB/s | 264 kB 00:00 2024-04-25T01:19:21.986 INFO:teuthology.orchestra.run.smithi077.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 6.9 MB/s | 762 kB 00:00 2024-04-25T01:19:22.053 INFO:teuthology.orchestra.run.smithi077.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 274 kB/s | 18 kB 00:00 2024-04-25T01:19:22.186 INFO:teuthology.orchestra.run.smithi077.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 603 kB/s | 80 kB 00:00 2024-04-25T01:19:22.378 INFO:teuthology.orchestra.run.smithi077.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2024-04-25T01:19:22.529 INFO:teuthology.orchestra.run.smithi077.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-25T01:19:22.688 INFO:teuthology.orchestra.run.smithi077.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-04-25T01:19:22.963 INFO:teuthology.orchestra.run.smithi077.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 4.5 MB/s | 1.2 MB 00:00 2024-04-25T01:19:23.196 INFO:teuthology.orchestra.run.smithi077.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 457 kB/s | 106 kB 00:00 2024-04-25T01:19:23.363 INFO:teuthology.orchestra.run.smithi077.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 812 kB/s | 135 kB 00:00 2024-04-25T01:19:23.770 INFO:teuthology.orchestra.run.smithi121.stdout:(10/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 2.4 MB/s | 10 MB 00:04 2024-04-25T01:19:23.770 INFO:teuthology.orchestra.run.smithi077.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 881 kB/s | 125 kB 00:00 2024-04-25T01:19:23.770 INFO:teuthology.orchestra.run.smithi077.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.2 MB/s | 215 kB 00:00 2024-04-25T01:19:23.780 INFO:teuthology.orchestra.run.smithi121.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 1.6 MB/s | 222 kB 00:00 2024-04-25T01:19:23.846 INFO:teuthology.orchestra.run.smithi121.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 406 kB/s | 27 kB 00:00 2024-04-25T01:19:23.930 INFO:teuthology.orchestra.run.smithi121.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 380 kB/s | 32 kB 00:00 2024-04-25T01:19:24.264 INFO:teuthology.orchestra.run.smithi077.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 1.2 MB/s | 689 kB 00:00 2024-04-25T01:19:24.364 INFO:teuthology.orchestra.run.smithi121.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 1.7 MB/s | 745 kB 00:00 2024-04-25T01:19:24.547 INFO:teuthology.orchestra.run.smithi121.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 892 kB/s | 163 kB 00:00 2024-04-25T01:19:24.639 INFO:teuthology.orchestra.run.smithi121.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.4 MB/s | 127 kB 00:00 2024-04-25T01:19:24.656 INFO:teuthology.orchestra.run.smithi077.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 2.4 MB/s | 943 kB 00:00 2024-04-25T01:19:24.873 INFO:teuthology.orchestra.run.smithi077.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 492 kB/s | 107 kB 00:00 2024-04-25T01:19:24.923 INFO:teuthology.orchestra.run.smithi121.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 1.7 MB/s | 501 kB 00:00 2024-04-25T01:19:25.007 INFO:teuthology.orchestra.run.smithi077.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 161 kB/s | 21 kB 00:00 2024-04-25T01:19:25.382 INFO:teuthology.orchestra.run.smithi077.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 836 kB/s | 313 kB 00:00 2024-04-25T01:19:25.566 INFO:teuthology.orchestra.run.smithi077.stdout:(52/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 1.1 MB/s | 10 MB 00:09 2024-04-25T01:19:25.582 INFO:teuthology.orchestra.run.smithi077.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 163 kB/s | 33 kB 00:00 2024-04-25T01:19:25.708 INFO:teuthology.orchestra.run.smithi077.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 146 kB/s | 18 kB 00:00 2024-04-25T01:19:25.808 INFO:teuthology.orchestra.run.smithi077.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-04-25T01:19:26.058 INFO:teuthology.orchestra.run.smithi077.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 642 kB/s | 160 kB 00:00 2024-04-25T01:19:26.142 INFO:teuthology.orchestra.run.smithi077.stdout:(57/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.2 MB/s | 3.0 MB 00:00 2024-04-25T01:19:26.292 INFO:teuthology.orchestra.run.smithi077.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 193 kB/s | 45 kB 00:00 2024-04-25T01:19:26.576 INFO:teuthology.orchestra.run.smithi077.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 1.5 MB/s | 662 kB 00:00 2024-04-25T01:19:26.726 INFO:teuthology.orchestra.run.smithi077.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 1.3 MB/s | 582 kB 00:00 2024-04-25T01:19:26.752 INFO:teuthology.orchestra.run.smithi077.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 145 kB/s | 25 kB 00:00 2024-04-25T01:19:26.935 INFO:teuthology.orchestra.run.smithi077.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 227 kB/s | 47 kB 00:00 2024-04-25T01:19:27.086 INFO:teuthology.orchestra.run.smithi077.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 876 kB/s | 292 kB 00:00 2024-04-25T01:19:27.244 INFO:teuthology.orchestra.run.smithi077.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 222 kB/s | 35 kB 00:00 2024-04-25T01:19:27.290 INFO:teuthology.orchestra.run.smithi121.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 2.2 MB/s | 5.3 MB 00:02 2024-04-25T01:19:27.295 INFO:teuthology.orchestra.run.smithi077.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 525 kB/s | 188 kB 00:00 2024-04-25T01:19:27.415 INFO:teuthology.orchestra.run.smithi121.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 362 kB/s | 45 kB 00:00 2024-04-25T01:19:27.523 INFO:teuthology.orchestra.run.smithi121.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.2 MB/s | 133 kB 00:00 2024-04-25T01:19:27.640 INFO:teuthology.orchestra.run.smithi121.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.4 MB/s | 162 kB 00:00 2024-04-25T01:19:27.807 INFO:teuthology.orchestra.run.smithi121.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 1.9 MB/s | 321 kB 00:00 2024-04-25T01:19:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 1.8 MB/s | 299 kB 00:00 2024-04-25T01:19:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.1 MB/s | 99 kB 00:00 2024-04-25T01:19:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.0 MB/s | 85 kB 00:00 2024-04-25T01:19:28.486 INFO:teuthology.orchestra.run.smithi077.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 3.8 MB/s | 4.7 MB 00:01 2024-04-25T01:19:28.870 INFO:teuthology.orchestra.run.smithi077.stdout:(67/152): ceph-mgr-diskprediction-local-19.0.0- 1.0 MB/s | 7.4 MB 00:07 2024-04-25T01:19:29.112 INFO:teuthology.orchestra.run.smithi077.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 2.4 MB/s | 4.4 MB 00:01 2024-04-25T01:19:29.337 INFO:teuthology.orchestra.run.smithi077.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-04-25T01:19:29.362 INFO:teuthology.orchestra.run.smithi077.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 533 kB/s | 133 kB 00:00 2024-04-25T01:19:29.408 INFO:teuthology.orchestra.run.smithi121.stdout:(26/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 2.5 MB/s | 3.1 MB 00:01 2024-04-25T01:19:29.483 INFO:teuthology.orchestra.run.smithi121.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.2 MB/s | 171 kB 00:00 2024-04-25T01:19:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 423 kB/s | 32 kB 00:00 2024-04-25T01:19:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.8 MB/s | 141 kB 00:00 2024-04-25T01:19:29.788 INFO:teuthology.orchestra.run.smithi077.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 993 kB/s | 911 kB 00:00 2024-04-25T01:19:29.996 INFO:teuthology.orchestra.run.smithi077.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 420 kB/s | 87 kB 00:00 2024-04-25T01:19:30.146 INFO:teuthology.orchestra.run.smithi077.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 1.3 MB/s | 1.0 MB 00:00 2024-04-25T01:19:30.413 INFO:teuthology.orchestra.run.smithi077.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 916 kB/s | 244 kB 00:00 2024-04-25T01:19:30.739 INFO:teuthology.orchestra.run.smithi077.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 763 kB/s | 248 kB 00:00 2024-04-25T01:19:30.897 INFO:teuthology.orchestra.run.smithi077.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 162 kB/s | 26 kB 00:00 2024-04-25T01:19:31.040 INFO:teuthology.orchestra.run.smithi077.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 135 kB/s | 19 kB 00:00 2024-04-25T01:19:31.251 INFO:teuthology.orchestra.run.smithi121.stdout:(30/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 4.3 MB/s | 48 MB 00:11 2024-04-25T01:19:31.290 INFO:teuthology.orchestra.run.smithi077.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 689 kB/s | 172 kB 00:00 2024-04-25T01:19:31.309 INFO:teuthology.orchestra.run.smithi121.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 2.1 MB/s | 3.5 MB 00:01 2024-04-25T01:19:31.440 INFO:teuthology.orchestra.run.smithi077.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 231 kB/s | 35 kB 00:00 2024-04-25T01:19:31.485 INFO:teuthology.orchestra.run.smithi121.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 1.4 MB/s | 249 kB 00:00 2024-04-25T01:19:31.577 INFO:teuthology.orchestra.run.smithi121.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 540 kB/s | 49 kB 00:00 2024-04-25T01:19:31.644 INFO:teuthology.orchestra.run.smithi121.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 229 kB/s | 15 kB 00:00 2024-04-25T01:19:31.761 INFO:teuthology.orchestra.run.smithi121.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 2.2 MB/s | 264 kB 00:00 2024-04-25T01:19:31.969 INFO:teuthology.orchestra.run.smithi121.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 3.6 MB/s | 762 kB 00:00 2024-04-25T01:19:32.028 INFO:teuthology.orchestra.run.smithi121.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 314 kB/s | 18 kB 00:00 2024-04-25T01:19:32.370 INFO:teuthology.orchestra.run.smithi121.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 235 kB/s | 80 kB 00:00 2024-04-25T01:19:32.690 INFO:teuthology.orchestra.run.smithi077.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 1.7 MB/s | 5.8 MB 00:03 2024-04-25T01:19:32.770 INFO:teuthology.orchestra.run.smithi121.stdout:(39/152): ceph-mgr-diskprediction-local-19.0.0- 4.9 MB/s | 7.4 MB 00:01 2024-04-25T01:19:32.904 INFO:teuthology.orchestra.run.smithi121.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-04-25T01:19:32.982 INFO:teuthology.orchestra.run.smithi077.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 860 kB/s | 250 kB 00:00 2024-04-25T01:19:33.046 INFO:teuthology.orchestra.run.smithi121.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-04-25T01:19:33.263 INFO:teuthology.orchestra.run.smithi121.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 5.7 MB/s | 1.2 MB 00:00 2024-04-25T01:19:33.316 INFO:teuthology.orchestra.run.smithi077.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.7 MB/s | 5.1 MB 00:01 2024-04-25T01:19:33.341 INFO:teuthology.orchestra.run.smithi077.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 620 kB/s | 222 kB 00:00 2024-04-25T01:19:33.388 INFO:teuthology.orchestra.run.smithi121.stdout:(43/152): libquadmath-11.4.1-3.el9.x86_64.rpm 316 kB/s | 195 kB 00:00 2024-04-25T01:19:33.408 INFO:teuthology.orchestra.run.smithi077.stdout:(84/152): python3-babel-2.9.1-2.el9.noarch.rpm 1.7 MB/s | 6.0 MB 00:03 2024-04-25T01:19:33.413 INFO:teuthology.orchestra.run.smithi121.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 709 kB/s | 106 kB 00:00 2024-04-25T01:19:33.497 INFO:teuthology.orchestra.run.smithi121.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-04-25T01:19:33.522 INFO:teuthology.orchestra.run.smithi121.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-04-25T01:19:33.542 INFO:teuthology.orchestra.run.smithi077.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 211 kB/s | 42 kB 00:00 2024-04-25T01:19:33.567 INFO:teuthology.orchestra.run.smithi077.stdout:(86/152): python3-packaging-20.9-5.el9.noarch.r 308 kB/s | 77 kB 00:00 2024-04-25T01:19:33.614 INFO:teuthology.orchestra.run.smithi121.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-04-25T01:19:33.792 INFO:teuthology.orchestra.run.smithi077.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 702 kB/s | 269 kB 00:00 2024-04-25T01:19:33.826 INFO:teuthology.orchestra.run.smithi077.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 559 kB/s | 159 kB 00:00 2024-04-25T01:19:33.865 INFO:teuthology.orchestra.run.smithi121.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 3.7 MB/s | 943 kB 00:00 2024-04-25T01:19:33.915 INFO:teuthology.orchestra.run.smithi121.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-04-25T01:19:33.941 INFO:teuthology.orchestra.run.smithi121.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 689 kB 00:00 2024-04-25T01:19:33.951 INFO:teuthology.orchestra.run.smithi077.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 726 kB/s | 279 kB 00:00 2024-04-25T01:19:33.966 INFO:teuthology.orchestra.run.smithi121.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-25T01:19:33.976 INFO:teuthology.orchestra.run.smithi077.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 277 kB/s | 51 kB 00:00 2024-04-25T01:19:34.010 INFO:teuthology.orchestra.run.smithi077.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 292 kB/s | 54 kB 00:00 2024-04-25T01:19:34.015 INFO:teuthology.orchestra.run.smithi121.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 655 kB/s | 33 kB 00:00 2024-04-25T01:19:34.227 INFO:teuthology.orchestra.run.smithi077.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 166 kB/s | 42 kB 00:00 2024-04-25T01:19:34.252 INFO:teuthology.orchestra.run.smithi077.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 182 kB/s | 44 kB 00:00 2024-04-25T01:19:34.266 INFO:teuthology.orchestra.run.smithi121.stdout:(53/152): ceph-common-19.0.0-2445.gd2c45d97.el9 1.1 MB/s | 21 MB 00:19 2024-04-25T01:19:34.291 INFO:teuthology.orchestra.run.smithi121.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 894 kB/s | 313 kB 00:00 2024-04-25T01:19:34.358 INFO:teuthology.orchestra.run.smithi121.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.7 MB/s | 3.0 MB 00:00 2024-04-25T01:19:34.383 INFO:teuthology.orchestra.run.smithi121.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-04-25T01:19:34.409 INFO:teuthology.orchestra.run.smithi121.stdout:(57/152): flexiblas-openblas-openmp-3.0.4-8.el9 128 kB/s | 18 kB 00:00 2024-04-25T01:19:34.411 INFO:teuthology.orchestra.run.smithi077.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 226 kB/s | 36 kB 00:00 2024-04-25T01:19:34.459 INFO:teuthology.orchestra.run.smithi121.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 600 kB/s | 45 kB 00:00 2024-04-25T01:19:34.534 INFO:teuthology.orchestra.run.smithi121.stdout:(59/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 911 kB/s | 160 kB 00:00 2024-04-25T01:19:34.559 INFO:teuthology.orchestra.run.smithi121.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 582 kB 00:00 2024-04-25T01:19:34.618 INFO:teuthology.orchestra.run.smithi121.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 305 kB/s | 25 kB 00:00 2024-04-25T01:19:34.643 INFO:teuthology.orchestra.run.smithi121.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 568 kB/s | 47 kB 00:00 2024-04-25T01:19:34.661 INFO:teuthology.orchestra.run.smithi077.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 509 kB/s | 221 kB 00:00 2024-04-25T01:19:34.710 INFO:teuthology.orchestra.run.smithi121.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-25T01:19:34.711 INFO:teuthology.orchestra.run.smithi077.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 158 kB/s | 47 kB 00:00 2024-04-25T01:19:34.777 INFO:teuthology.orchestra.run.smithi121.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 526 kB/s | 35 kB 00:00 2024-04-25T01:19:34.853 INFO:teuthology.orchestra.run.smithi121.stdout:(65/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.2 MB/s | 292 kB 00:00 2024-04-25T01:19:34.911 INFO:teuthology.orchestra.run.smithi077.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 289 kB/s | 58 kB 00:00 2024-04-25T01:19:35.103 INFO:teuthology.orchestra.run.smithi121.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-04-25T01:19:35.128 INFO:teuthology.orchestra.run.smithi121.stdout:(67/152): librdkafka-1.6.1-102.el9.x86_64.rpm 921 kB/s | 662 kB 00:00 2024-04-25T01:19:35.187 INFO:teuthology.orchestra.run.smithi077.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 580 kB/s | 305 kB 00:00 2024-04-25T01:19:35.212 INFO:teuthology.orchestra.run.smithi077.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 312 kB/s | 94 kB 00:00 2024-04-25T01:19:35.228 INFO:teuthology.orchestra.run.smithi121.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-25T01:19:35.279 INFO:teuthology.orchestra.run.smithi121.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-04-25T01:19:35.412 INFO:teuthology.orchestra.run.smithi077.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 224 kB/s | 45 kB 00:00 2024-04-25T01:19:35.438 INFO:teuthology.orchestra.run.smithi077.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 254 kB/s | 64 kB 00:00 2024-04-25T01:19:35.654 INFO:teuthology.orchestra.run.smithi121.stdout:(70/152): postgresql-server-13.14-1.el9.x86_64. 15 MB/s | 5.8 MB 00:00 2024-04-25T01:19:35.687 INFO:teuthology.orchestra.run.smithi121.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 1.6 MB/s | 911 kB 00:00 2024-04-25T01:19:35.729 INFO:teuthology.orchestra.run.smithi121.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-04-25T01:19:35.780 INFO:teuthology.orchestra.run.smithi077.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 61 kB/s | 22 kB 00:00 2024-04-25T01:19:35.821 INFO:teuthology.orchestra.run.smithi121.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 655 kB/s | 87 kB 00:00 2024-04-25T01:19:36.238 INFO:teuthology.orchestra.run.smithi121.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 586 kB/s | 244 kB 00:00 2024-04-25T01:19:36.313 INFO:teuthology.orchestra.run.smithi121.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-04-25T01:19:36.463 INFO:teuthology.orchestra.run.smithi077.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 842 kB/s | 863 kB 00:01 2024-04-25T01:19:36.564 INFO:teuthology.orchestra.run.smithi121.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 763 kB/s | 248 kB 00:00 2024-04-25T01:19:36.574 INFO:teuthology.orchestra.run.smithi077.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 692 kB/s | 548 kB 00:00 2024-04-25T01:19:36.589 INFO:teuthology.orchestra.run.smithi121.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 93 kB/s | 26 kB 00:00 2024-04-25T01:19:36.614 INFO:teuthology.orchestra.run.smithi121.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-04-25T01:19:36.664 INFO:teuthology.orchestra.run.smithi121.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 696 kB/s | 35 kB 00:00 2024-04-25T01:19:36.697 INFO:teuthology.orchestra.run.smithi077.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-04-25T01:19:36.723 INFO:teuthology.orchestra.run.smithi077.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 129 kB/s | 19 kB 00:00 2024-04-25T01:19:36.790 INFO:teuthology.orchestra.run.smithi121.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 858 kB/s | 172 kB 00:00 2024-04-25T01:19:37.123 INFO:teuthology.orchestra.run.smithi121.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 752 kB/s | 250 kB 00:00 2024-04-25T01:19:37.190 INFO:teuthology.orchestra.run.smithi121.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.7 MB/s | 5.1 MB 00:00 2024-04-25T01:19:37.240 INFO:teuthology.orchestra.run.smithi121.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T01:19:37.324 INFO:teuthology.orchestra.run.smithi121.stdout:(84/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 1.8 MB/s | 4.4 MB 00:02 2024-04-25T01:19:37.351 INFO:teuthology.orchestra.run.smithi121.stdout:(85/152): python3-oauthlib-3.1.1-5.el9.noarch.r 984 kB/s | 222 kB 00:00 2024-04-25T01:19:37.375 INFO:teuthology.orchestra.run.smithi121.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 316 kB/s | 42 kB 00:00 2024-04-25T01:19:37.441 INFO:teuthology.orchestra.run.smithi121.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-04-25T01:19:37.508 INFO:teuthology.orchestra.run.smithi121.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 765 kB/s | 51 kB 00:00 2024-04-25T01:19:37.534 INFO:teuthology.orchestra.run.smithi121.stdout:(89/152): python3-protobuf-3.14.0-13.el9.noarch 1.3 MB/s | 269 kB 00:00 2024-04-25T01:19:37.550 INFO:teuthology.orchestra.run.smithi121.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-04-25T01:19:37.593 INFO:teuthology.orchestra.run.smithi121.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 998 kB/s | 42 kB 00:00 2024-04-25T01:19:37.618 INFO:teuthology.orchestra.run.smithi121.stdout:(92/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.1 MB/s | 279 kB 00:00 2024-04-25T01:19:37.635 INFO:teuthology.orchestra.run.smithi121.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-04-25T01:19:37.687 INFO:teuthology.orchestra.run.smithi121.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 688 kB/s | 36 kB 00:00 2024-04-25T01:19:37.737 INFO:teuthology.orchestra.run.smithi121.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 950 kB/s | 47 kB 00:00 2024-04-25T01:19:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:(96/152): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-04-25T01:19:37.788 INFO:teuthology.orchestra.run.smithi121.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-25T01:19:37.815 INFO:teuthology.orchestra.run.smithi077.stdout:(107/152): libarrow-9.0.0-10.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-04-25T01:19:37.840 INFO:teuthology.orchestra.run.smithi077.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 23 kB/s | 26 kB 00:01 2024-04-25T01:19:37.846 INFO:teuthology.orchestra.run.smithi121.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-25T01:19:37.872 INFO:teuthology.orchestra.run.smithi121.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 531 kB/s | 58 kB 00:00 2024-04-25T01:19:37.882 INFO:teuthology.orchestra.run.smithi077.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 730 kB/s | 49 kB 00:00 2024-04-25T01:19:37.897 INFO:teuthology.orchestra.run.smithi121.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T01:19:37.907 INFO:teuthology.orchestra.run.smithi077.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-25T01:19:37.922 INFO:teuthology.orchestra.run.smithi121.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 891 kB/s | 45 kB 00:00 2024-04-25T01:19:37.949 INFO:teuthology.orchestra.run.smithi077.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-25T01:19:38.014 INFO:teuthology.orchestra.run.smithi121.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-04-25T01:19:38.058 INFO:teuthology.orchestra.run.smithi077.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.4 MB/s | 838 kB 00:00 2024-04-25T01:19:38.123 INFO:teuthology.orchestra.run.smithi121.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 4.2 MB/s | 863 kB 00:00 2024-04-25T01:19:38.124 INFO:teuthology.orchestra.run.smithi077.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 3.1 MB/s | 548 kB 00:00 2024-04-25T01:19:38.150 INFO:teuthology.orchestra.run.smithi077.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 476 kB/s | 43 kB 00:00 2024-04-25T01:19:38.200 INFO:teuthology.orchestra.run.smithi077.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-04-25T01:19:38.225 INFO:teuthology.orchestra.run.smithi077.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-04-25T01:19:38.284 INFO:teuthology.orchestra.run.smithi077.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-25T01:19:38.334 INFO:teuthology.orchestra.run.smithi077.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-04-25T01:19:38.368 INFO:teuthology.orchestra.run.smithi077.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-04-25T01:19:38.422 INFO:teuthology.orchestra.run.smithi077.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-04-25T01:19:38.469 INFO:teuthology.orchestra.run.smithi077.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-25T01:19:38.719 INFO:teuthology.orchestra.run.smithi077.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 6.8 MB/s | 2.0 MB 00:00 2024-04-25T01:19:38.744 INFO:teuthology.orchestra.run.smithi077.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 525 kB/s | 144 kB 00:00 2024-04-25T01:19:38.765 INFO:teuthology.orchestra.run.smithi121.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 481 kB/s | 309 kB 00:00 2024-04-25T01:19:38.811 INFO:teuthology.orchestra.run.smithi077.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 117 kB/s | 11 kB 00:00 2024-04-25T01:19:38.828 INFO:teuthology.orchestra.run.smithi077.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-04-25T01:19:38.887 INFO:teuthology.orchestra.run.smithi077.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 308 kB/s | 23 kB 00:00 2024-04-25T01:19:38.890 INFO:teuthology.orchestra.run.smithi121.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 156 kB/s | 19 kB 00:00 2024-04-25T01:19:38.912 INFO:teuthology.orchestra.run.smithi077.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-04-25T01:19:38.924 INFO:teuthology.orchestra.run.smithi121.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 603 kB/s | 548 kB 00:00 2024-04-25T01:19:38.971 INFO:teuthology.orchestra.run.smithi077.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-04-25T01:19:39.057 INFO:teuthology.orchestra.run.smithi121.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-04-25T01:19:39.088 INFO:teuthology.orchestra.run.smithi077.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-04-25T01:19:39.113 INFO:teuthology.orchestra.run.smithi077.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 326 kB/s | 46 kB 00:00 2024-04-25T01:19:39.173 INFO:teuthology.orchestra.run.smithi077.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 943 kB/s | 79 kB 00:00 2024-04-25T01:19:39.184 INFO:teuthology.orchestra.run.smithi121.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-04-25T01:19:39.200 INFO:teuthology.orchestra.run.smithi077.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 672 kB/s | 58 kB 00:00 2024-04-25T01:19:39.258 INFO:teuthology.orchestra.run.smithi077.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.1 MB/s | 272 kB 00:00 2024-04-25T01:19:39.283 INFO:teuthology.orchestra.run.smithi077.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-04-25T01:19:39.324 INFO:teuthology.orchestra.run.smithi121.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 475 kB/s | 67 kB 00:00 2024-04-25T01:19:39.341 INFO:teuthology.orchestra.run.smithi077.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-04-25T01:19:39.367 INFO:teuthology.orchestra.run.smithi077.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-04-25T01:19:39.409 INFO:teuthology.orchestra.run.smithi077.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 460 kB/s | 31 kB 00:00 2024-04-25T01:19:39.442 INFO:teuthology.orchestra.run.smithi077.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-25T01:19:39.493 INFO:teuthology.orchestra.run.smithi077.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 708 kB/s | 59 kB 00:00 2024-04-25T01:19:39.525 INFO:teuthology.orchestra.run.smithi121.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 755 kB/s | 151 kB 00:00 2024-04-25T01:19:39.526 INFO:teuthology.orchestra.run.smithi077.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2024-04-25T01:19:39.585 INFO:teuthology.orchestra.run.smithi077.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 555 kB/s | 51 kB 00:00 2024-04-25T01:19:39.619 INFO:teuthology.orchestra.run.smithi077.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-25T01:19:39.661 INFO:teuthology.orchestra.run.smithi077.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-25T01:19:39.719 INFO:teuthology.orchestra.run.smithi077.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-04-25T01:19:39.744 INFO:teuthology.orchestra.run.smithi077.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-04-25T01:19:39.812 INFO:teuthology.orchestra.run.smithi077.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-25T01:19:40.020 INFO:teuthology.orchestra.run.smithi077.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 5.7 MB/s | 1.6 MB 00:00 2024-04-25T01:19:40.108 INFO:teuthology.orchestra.run.smithi121.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.4 MB/s | 838 kB 00:00 2024-04-25T01:19:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 1.6 MB/s | 548 kB 00:00 2024-04-25T01:19:40.559 INFO:teuthology.orchestra.run.smithi121.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-04-25T01:19:40.659 INFO:teuthology.orchestra.run.smithi121.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 322 kB/s | 32 kB 00:00 2024-04-25T01:19:40.793 INFO:teuthology.orchestra.run.smithi121.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 112 kB/s | 15 kB 00:00 2024-04-25T01:19:40.935 INFO:teuthology.orchestra.run.smithi121.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.2 MB/s | 172 kB 00:00 2024-04-25T01:19:41.185 INFO:teuthology.orchestra.run.smithi121.stdout:(117/152): python3-cherrypy-18.6.1-2.el9.noarch 1.4 MB/s | 358 kB 00:00 2024-04-25T01:19:41.452 INFO:teuthology.orchestra.run.smithi121.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.1 MB/s | 303 kB 00:00 2024-04-25T01:19:41.453 INFO:teuthology.orchestra.run.smithi077.stdout:(148/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 2.2 MB/s | 3.1 MB 00:01 2024-04-25T01:19:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 264 kB/s | 35 kB 00:00 2024-04-25T01:19:41.604 INFO:teuthology.orchestra.run.smithi077.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 9.0 MB/s | 1.3 MB 00:00 2024-04-25T01:19:41.719 INFO:teuthology.orchestra.run.smithi121.stdout:(120/152): python3-google-auth-2.29.0-1.el9.noa 1.5 MB/s | 206 kB 00:00 2024-04-25T01:19:41.754 INFO:teuthology.orchestra.run.smithi077.stdout:(150/152): librados2-19.0.0-2445.gd2c45d97.el9. 1.7 MB/s | 3.4 MB 00:01 2024-04-25T01:19:41.811 INFO:teuthology.orchestra.run.smithi121.stdout:(121/152): libarrow-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 4.4 MB 00:02 2024-04-25T01:19:41.813 INFO:teuthology.orchestra.run.smithi077.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-04-25T01:19:42.028 INFO:teuthology.orchestra.run.smithi121.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 667 kB/s | 144 kB 00:00 2024-04-25T01:19:42.137 INFO:teuthology.orchestra.run.smithi121.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-04-25T01:19:42.229 INFO:teuthology.orchestra.run.smithi121.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-04-25T01:19:42.354 INFO:teuthology.orchestra.run.smithi121.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 185 kB/s | 23 kB 00:00 2024-04-25T01:19:42.488 INFO:teuthology.orchestra.run.smithi121.stdout:(126/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.7 MB/s | 2.0 MB 00:00 2024-04-25T01:19:42.514 INFO:teuthology.orchestra.run.smithi121.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 122 kB/s | 19 kB 00:00 2024-04-25T01:19:42.597 INFO:teuthology.orchestra.run.smithi121.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-04-25T01:19:42.722 INFO:teuthology.orchestra.run.smithi121.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 370 kB/s | 46 kB 00:00 2024-04-25T01:19:42.839 INFO:teuthology.orchestra.run.smithi121.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 676 kB/s | 79 kB 00:00 2024-04-25T01:19:42.964 INFO:teuthology.orchestra.run.smithi121.stdout:(131/152): python3-natsort-7.1.1-5.el9.noarch.r 462 kB/s | 58 kB 00:00 2024-04-25T01:19:43.014 INFO:teuthology.orchestra.run.smithi121.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 2.0 MB/s | 1.0 MB 00:00 2024-04-25T01:19:43.163 INFO:teuthology.orchestra.run.smithi077.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.8 MB/s | 16 MB 00:09 2024-04-25T01:19:43.165 INFO:teuthology.orchestra.run.smithi121.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-04-25T01:19:43.168 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:19:43.169 INFO:teuthology.orchestra.run.smithi077.stdout:Total 7.3 MB/s | 220 MB 00:30 2024-04-25T01:19:43.190 INFO:teuthology.orchestra.run.smithi121.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 94 kB/s | 16 kB 00:00 2024-04-25T01:19:43.360 INFO:teuthology.orchestra.run.smithi121.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 460 kB/s | 90 kB 00:00 2024-04-25T01:19:43.398 INFO:teuthology.orchestra.run.smithi121.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 135 kB/s | 28 kB 00:00 2024-04-25T01:19:43.653 INFO:teuthology.orchestra.run.smithi121.stdout:(137/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.7 MB/s | 16 MB 00:06 2024-04-25T01:19:43.799 INFO:teuthology.orchestra.run.smithi121.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 70 kB/s | 31 kB 00:00 2024-04-25T01:19:43.825 INFO:teuthology.orchestra.run.smithi121.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 443 kB/s | 188 kB 00:00 2024-04-25T01:19:43.900 INFO:teuthology.orchestra.run.smithi121.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-04-25T01:19:43.934 INFO:teuthology.orchestra.run.smithi121.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 470 kB/s | 51 kB 00:00 2024-04-25T01:19:44.050 INFO:teuthology.orchestra.run.smithi121.stdout:(142/152): python3-websocket-client-1.2.3-2.el9 769 kB/s | 90 kB 00:00 2024-04-25T01:19:44.084 INFO:teuthology.orchestra.run.smithi121.stdout:(143/152): python3-rsa-4.9-2.el9.noarch.rpm 137 kB/s | 59 kB 00:00 2024-04-25T01:19:44.109 INFO:teuthology.orchestra.run.smithi121.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-04-25T01:19:44.201 INFO:teuthology.orchestra.run.smithi121.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-04-25T01:19:44.268 INFO:teuthology.orchestra.run.smithi121.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-25T01:19:44.284 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-04-25T01:19:44.293 INFO:teuthology.orchestra.run.smithi121.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 1.7 MB/s | 427 kB 00:00 2024-04-25T01:19:44.429 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-04-25T01:19:44.429 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-04-25T01:19:44.652 INFO:teuthology.orchestra.run.smithi121.stdout:(148/152): librados2-19.0.0-2445.gd2c45d97.el9. 8.8 MB/s | 3.4 MB 00:00 2024-04-25T01:19:44.802 INFO:teuthology.orchestra.run.smithi121.stdout:(149/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 6.2 MB/s | 3.1 MB 00:00 2024-04-25T01:19:46.187 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-04-25T01:19:46.187 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-04-25T01:19:46.203 INFO:teuthology.orchestra.run.smithi121.stdout:(150/152): thrift-0.15.0-2.el9.x86_64.rpm 809 kB/s | 1.6 MB 00:02 2024-04-25T01:19:46.344 INFO:teuthology.orchestra.run.smithi121.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 1.6 MB/s | 2.4 MB 00:01 2024-04-25T01:19:46.378 INFO:teuthology.orchestra.run.smithi121.stdout:(152/152): openssl-3.2.1-1.el9.x86_64.rpm 801 kB/s | 1.3 MB 00:01 2024-04-25T01:19:46.381 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:19:46.381 INFO:teuthology.orchestra.run.smithi121.stdout:Total 6.9 MB/s | 220 MB 00:31 2024-04-25T01:19:47.478 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-25T01:19:47.621 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-25T01:19:47.621 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-25T01:19:49.385 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-25T01:19:49.385 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-25T01:19:52.155 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-04-25T01:19:52.242 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:19:52.280 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:19:52.334 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:19:52.368 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:19:52.393 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:19:52.436 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:19:52.840 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:19:52.886 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:19:53.032 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:19:53.062 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:19:53.152 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:19:53.191 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:19:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:19:53.389 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:19:53.416 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:19:53.519 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:19:53.543 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:19:53.605 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:19:53.627 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:19:53.650 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:19:53.827 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:19:53.860 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:19:53.966 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:19:54.063 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:19:54.101 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:19:54.135 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:19:54.200 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:19:54.243 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:19:54.286 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:19:54.319 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:19:54.347 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:19:54.415 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:19:54.469 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:19:54.748 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:19:54.748 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:19:54.748 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:19:54.748 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:19:54.748 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:19:54.749 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:19:54.790 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:19:54.841 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:19:54.917 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:19:55.048 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:19:55.103 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:19:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-25T01:19:55.489 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:19:55.527 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:19:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:19:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:19:55.620 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:19:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:19:55.683 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:19:55.698 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:19:55.727 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:19:55.761 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:19:55.800 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:19:55.842 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:19:55.986 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:19:56.012 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:19:56.036 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:19:56.091 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:19:56.133 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:19:56.172 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:19:56.203 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:19:56.240 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:19:56.253 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:19:56.276 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:19:56.286 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:19:56.323 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:19:56.366 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:19:56.366 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:19:56.405 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:19:56.408 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:19:56.543 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:19:56.550 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:19:56.591 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:19:56.601 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:19:56.619 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:19:56.630 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:19:56.661 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:19:56.750 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:19:56.754 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:19:56.772 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:19:56.786 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:19:56.826 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:19:56.834 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:19:56.852 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:19:56.856 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:19:56.876 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:19:56.880 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:19:56.951 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:19:57.073 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:19:57.079 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:19:57.106 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:19:57.109 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:19:57.128 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:19:57.163 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:19:57.202 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:19:57.217 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:19:57.234 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:19:57.264 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:19:57.298 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:19:57.308 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:19:57.332 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:19:57.342 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:19:57.373 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:19:57.374 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:19:57.408 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:19:57.447 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:19:57.481 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:19:57.494 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:19:57.526 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:19:57.538 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:19:57.558 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:19:57.569 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:19:57.585 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:19:57.624 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:19:57.653 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:19:57.707 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:19:57.732 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:19:57.755 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:19:57.788 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:19:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:19:57.874 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:19:57.909 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:19:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:19:57.981 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:19:58.050 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:19:58.096 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:19:58.169 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:19:58.313 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:19:58.373 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:19:58.503 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:19:58.552 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:19:58.586 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:19:58.692 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:19:58.744 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:19:58.831 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:19:58.871 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:19:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:19:58.969 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:19:58.998 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:19:59.041 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:19:59.051 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:19:59.077 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:19:59.079 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:19:59.122 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:19:59.155 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:19:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:19:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:19:59.412 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:19:59.484 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:19:59.516 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:19:59.553 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:19:59.599 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:19:59.635 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:19:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:19:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:19:59.800 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:19:59.855 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:19:59.909 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:19:59.927 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:19:59.940 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:20:00.006 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:20:00.100 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:20:00.142 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:20:00.180 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:20:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:20:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:20:00.303 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:20:00.431 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:20:00.463 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:20:00.499 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:20:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:20:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:20:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:20:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:20:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:20:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:20:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:20:00.778 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:20:00.864 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:20:00.908 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:20:00.939 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:20:00.986 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:20:01.085 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:20:01.134 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:20:01.141 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:20:01.216 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:20:01.243 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:20:01.277 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:20:01.590 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:20:01.623 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:20:01.714 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:20:01.746 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:20:01.781 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:20:01.854 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:20:01.896 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:20:01.930 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:20:02.029 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:20:02.080 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:20:02.168 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:20:02.207 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:20:02.259 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:20:02.293 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:20:02.364 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:20:02.386 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:20:02.395 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:20:02.414 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:20:02.491 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:20:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:20:03.236 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:20:04.443 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:20:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:20:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:20:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:20:04.598 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:04.633 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:04.898 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:20:04.967 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:20:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:20:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:20:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:20:05.178 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:05.212 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:20:05.301 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:20:05.302 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:20:05.302 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:20:05.302 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:05.338 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:20:05.604 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:20:05.628 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:20:05.699 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:20:05.730 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:20:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:20:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:20:08.756 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:20:08.756 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:20:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:08.789 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:20:16.773 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:20:16.773 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-04-25T01:20:16.773 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-04-25T01:20:16.773 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-04-25T01:20:16.773 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-04-25T01:20:16.773 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-04-25T01:20:16.773 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /root 2024-04-25T01:20:16.773 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-04-25T01:20:16.773 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:16.934 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:20:16.949 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:20:16.950 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:16.950 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:20:16.950 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:18.428 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:20:18.454 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:20:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:20:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:20:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:20:18.455 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:18.688 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:20:18.710 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:20:18.710 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:18.711 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:20:18.711 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:20:18.711 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:20:18.711 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:19.157 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:20:19.180 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:20:19.180 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:19.180 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:20:19.180 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:20:19.180 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:20:19.181 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:19.279 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:20:19.328 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:20:19.383 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:20:19.568 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:20:19.616 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:20:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:20:19.783 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:20:19.921 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:20:19.967 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:20:19.999 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:20:20.027 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:20:20.221 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:20:20.221 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-04-25T01:20:20.221 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-04-25T01:20:20.222 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-04-25T01:20:20.222 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-04-25T01:20:20.222 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-04-25T01:20:20.222 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-04-25T01:20:20.222 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-04-25T01:20:20.222 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:20.385 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:20:20.400 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:20:20.400 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:20.400 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:20:20.400 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:20.491 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:20:20.517 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:20:20.548 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:20:20.610 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:20:20.648 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:20:21.681 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:20:21.717 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:20:21.879 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:20:21.905 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:20:21.905 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:21.905 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:20:21.905 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:20:21.905 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:20:21.905 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:20:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:20:22.152 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:22.152 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:20:22.152 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:20:22.153 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:20:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:20:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:20:22.611 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:22.612 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:20:22.612 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:20:22.612 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:20:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:22.682 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:20:22.718 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:20:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:20:22.776 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:20:22.820 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:20:22.831 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:20:22.980 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:20:23.013 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:20:23.019 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:20:23.043 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:20:23.043 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:23.043 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:20:23.043 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:20:23.043 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:20:23.043 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:23.064 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:20:23.087 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:20:23.098 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:20:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:20:23.197 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:20:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:20:23.252 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:20:23.342 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:20:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:20:23.413 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:20:23.441 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:20:23.831 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:20:23.865 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:20:23.896 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:20:23.957 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:20:23.996 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:20:24.147 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:20:24.181 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:20:24.204 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:20:24.204 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:24.204 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:20:24.204 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:20:24.204 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:20:24.204 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:24.248 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:20:24.271 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:20:24.271 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:24.271 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:20:24.271 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:24.332 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:20:24.355 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:20:24.355 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:24.355 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:20:24.355 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:20:24.355 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:20:24.355 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:24.658 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:20:24.681 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:20:24.681 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:24.681 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:20:24.681 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:20:24.681 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:20:24.681 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:20:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:20:26.024 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:20:26.065 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:20:26.166 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:20:26.427 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:20:26.459 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:20:26.481 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:20:26.482 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:26.482 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:20:26.482 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:20:26.482 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:20:26.482 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:26.526 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:20:26.536 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:20:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:20:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:20:27.575 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:20:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:20:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:20:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:20:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:20:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:20:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:20:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:20:27.692 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:27.692 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:20:27.692 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:27.737 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:20:27.760 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:20:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:20:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:20:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:20:27.760 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:20:28.085 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:20:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:20:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:20:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:20:28.086 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:29.170 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:20:29.212 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:20:29.252 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:20:29.285 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:20:29.312 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:20:29.492 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:20:29.533 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:20:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:20:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:31.069 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:20:31.158 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:20:31.199 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:20:31.280 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:20:31.312 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:20:31.312 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:20:31.397 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:20:31.430 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:20:31.430 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:20:31.524 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:20:31.594 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:20:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:20:32.641 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:20:32.673 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:20:32.700 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:20:32.878 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:20:32.921 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:20:32.951 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:20:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:33.643 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:20:34.554 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:20:34.594 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:20:34.684 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:20:34.724 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:20:34.725 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:20:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:20:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:20:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:20:34.928 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:20:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:37.242 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:37.242 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:20:37.242 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:20:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:20:37.244 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:20:37.245 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:20:37.246 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:20:37.247 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:20:37.248 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:20:37.249 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:20:37.250 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:20:37.251 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:20:37.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:20:37.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:20:37.254 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:20:37.254 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:20:39.437 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:39.437 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:39.437 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-04-25T01:20:39.437 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.437 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.437 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:39.437 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:39.437 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-04-25T01:20:39.437 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:39.438 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:39.439 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:20:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:20:39.442 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:20:39.443 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:20:39.444 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:20:39.445 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:20:39.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:20:39.447 INFO:teuthology.orchestra.run.smithi077.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:20:39.448 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:20:39.448 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:20:39.448 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:20:39.448 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:39.448 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-04-25T01:20:39.840 DEBUG:teuthology.parallel:result is None 2024-04-25T01:20:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:20:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:20:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:20:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:20:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:20:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:20:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:20:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:20:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:20:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:20:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:20:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:20:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:20:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:20:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:20:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:20:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:20:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:20:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:20:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:20:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:20:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:20:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:20:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:20:42.968 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:42.968 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:42.968 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-04-25T01:20:42.968 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.968 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.968 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:42.968 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:42.968 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.969 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:42.970 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:20:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.972 INFO:teuthology.orchestra.run.smithi121.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:20:42.973 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:20:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:20:42.976 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.977 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:20:42.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:20:42.979 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:20:42.980 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:20:42.980 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:42.980 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-25T01:20:43.356 DEBUG:teuthology.parallel:result is None 2024-04-25T01:20:43.357 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:20:43.587 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:20:43.613 INFO:teuthology.orchestra.run.smithi077.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:20:43.614 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:20:43.614 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:20:43.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:20:43.842 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:20:43.863 INFO:teuthology.orchestra.run.smithi121.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:20:43.863 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:20:43.863 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:20:43.865 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T01:20:43.865 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:20:43.865 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:20:43.896 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:20:43.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:20:43.930 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T01:20:43.930 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:20:43.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:20:43.965 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:20:44.032 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:20:44.032 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:20:44.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:20:44.118 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T01:20:44.118 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:20:44.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:20:44.144 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:20:44.209 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:20:44.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:20:44.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:20:44.297 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T01:20:44.297 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:20:44.297 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:20:44.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:20:44.388 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:20:44.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:20:44.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:20:44.477 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T01:20:44.557 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T01:20:44.557 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /var/log/ceph 2024-04-25T01:20:44.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2024-04-25T01:20:44.582 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T01:20:44.583 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T01:20:44.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T01:20:44.647 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T01:20:44.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T01:20:44.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T01:20:44.720 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-25T01:20:44.720 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-25T01:20:44.720 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-25T01:20:44.720 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:20:44.721 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T01:20:44.769 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T01:20:44.876 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T01:20:44.917 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:20:44.917 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T01:20:44.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T01:20:45.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T01:20:45.083 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T01:20:45.083 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'force file system read-only', 'bad backtrace', 'MDS in read-only mode', '\\(MDS_READ_ONLY\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T01:20:45.083 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319', 'branch': 'squid', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/auto-repair}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7672319', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'force file system read-only', 'bad backtrace', 'MDS in read-only mode', '\\(MDS_READ_ONLY\\)'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 8528, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'sleep_before_teardown': 0, 'subset': '122/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi077.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF0BLrIbcXB1/3PN9Zx1XwhG5TqPqQIE/fkSgDgLQuYq8F3wbH5vKKhUlRH3Yz+h5Rwrrwflj7TOIQfGgDW5bYs=', 'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAexl1tvEKG02Tx7lY8UrCME3hEOnCIZCTBvU7BK2Md/vEyXCf3aMWUEqWanYfdYNqi8BRdrnacvNTb4cqEiqbc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'force file system read-only', 'bad backtrace', 'MDS in read-only mode', '\\(MDS_READ_ONLY\\)'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_auto_repair']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_21:24:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T01:20:45.084 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T01:20:45.086 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T01:20:45.132 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T01:20:45.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T01:20:45.195 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:20:45.195 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:20:45.219 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:20:45.220 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-04-25T01:20:45.274 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:20:45.275 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:20:45.275 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T01:20:45.275 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:20:45.275 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:20:45.275 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-25 01:20:32.013244237 +0000 2024-04-25T01:20:45.275 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-25 01:18:32.141703871 +0000 2024-04-25T01:20:45.275 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-25 01:18:32.141703871 +0000 2024-04-25T01:20:45.275 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-25 01:18:32.141703871 +0000 2024-04-25T01:20:45.275 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:20:45.337 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-25T01:20:45.337 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-25T01:20:45.337 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000209431 s, 2.4 MB/s 2024-04-25T01:20:45.338 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:20:45.395 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-04-25T01:20:45.449 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:20:45.449 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:20:45.449 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T01:20:45.449 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:20:45.450 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:20:45.450 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-25 01:20:32.014244217 +0000 2024-04-25T01:20:45.450 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-25 01:18:32.487696990 +0000 2024-04-25T01:20:45.450 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-25 01:18:32.487696990 +0000 2024-04-25T01:20:45.450 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-25 01:18:32.487696990 +0000 2024-04-25T01:20:45.450 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:20:45.511 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-25T01:20:45.511 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-25T01:20:45.511 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000188072 s, 2.7 MB/s 2024-04-25T01:20:45.512 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:20:45.567 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-04-25T01:20:45.622 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:20:45.622 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:20:45.622 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T01:20:45.622 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:20:45.622 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:20:45.622 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-25 01:20:32.014244217 +0000 2024-04-25T01:20:45.622 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-25 01:18:32.822690328 +0000 2024-04-25T01:20:45.622 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-25 01:18:32.823690308 +0000 2024-04-25T01:20:45.622 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-25 01:18:32.822690328 +0000 2024-04-25T01:20:45.622 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:20:45.684 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-25T01:20:45.684 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-25T01:20:45.684 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000205234 s, 2.5 MB/s 2024-04-25T01:20:45.685 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:20:45.740 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-04-25T01:20:45.795 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:20:45.795 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:20:45.795 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-25T01:20:45.796 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:20:45.796 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:20:45.796 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-25 01:20:32.015244197 +0000 2024-04-25T01:20:45.796 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-25 01:18:33.170683407 +0000 2024-04-25T01:20:45.796 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-25 01:18:33.170683407 +0000 2024-04-25T01:20:45.796 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-25 01:18:33.170683407 +0000 2024-04-25T01:20:45.796 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:20:45.858 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-25T01:20:45.858 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-25T01:20:45.858 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000231057 s, 2.2 MB/s 2024-04-25T01:20:45.859 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:20:45.914 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T01:20:45.914 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:20:45.914 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:20:45.931 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:20:45.931 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-04-25T01:20:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:20:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:20:45.986 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T01:20:45.986 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:20:45.986 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:20:45.986 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-25 01:20:35.411058019 +0000 2024-04-25T01:20:45.986 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-25 01:18:32.434454873 +0000 2024-04-25T01:20:45.986 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-25 01:18:32.434454873 +0000 2024-04-25T01:20:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-25 01:18:32.434454873 +0000 2024-04-25T01:20:45.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:20:46.049 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-25T01:20:46.049 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-25T01:20:46.049 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000290848 s, 1.8 MB/s 2024-04-25T01:20:46.050 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:20:46.107 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-04-25T01:20:46.162 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:20:46.162 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:20:46.162 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T01:20:46.162 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:20:46.163 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:20:46.163 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-25 01:20:35.411058019 +0000 2024-04-25T01:20:46.163 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-25 01:18:32.782449785 +0000 2024-04-25T01:20:46.163 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-25 01:18:32.782449785 +0000 2024-04-25T01:20:46.163 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-25 01:18:32.782449785 +0000 2024-04-25T01:20:46.163 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:20:46.225 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-25T01:20:46.225 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-25T01:20:46.225 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000195348 s, 2.6 MB/s 2024-04-25T01:20:46.226 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:20:46.282 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-04-25T01:20:46.337 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:20:46.337 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:20:46.337 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T01:20:46.337 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:20:46.337 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:20:46.338 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-25 01:20:35.412058005 +0000 2024-04-25T01:20:46.338 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-25 01:18:33.140444551 +0000 2024-04-25T01:20:46.338 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-25 01:18:33.140444551 +0000 2024-04-25T01:20:46.338 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-25 01:18:33.140444551 +0000 2024-04-25T01:20:46.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:20:46.399 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-25T01:20:46.399 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-25T01:20:46.399 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000205377 s, 2.5 MB/s 2024-04-25T01:20:46.400 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:20:46.455 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-04-25T01:20:46.510 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:20:46.510 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:20:46.510 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T01:20:46.510 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:20:46.510 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:20:46.511 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-25 01:20:35.412058005 +0000 2024-04-25T01:20:46.511 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-25 01:18:33.486439492 +0000 2024-04-25T01:20:46.511 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-25 01:18:33.486439492 +0000 2024-04-25T01:20:46.511 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-25 01:18:33.486439492 +0000 2024-04-25T01:20:46.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:20:46.572 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-25T01:20:46.572 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-25T01:20:46.572 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000233824 s, 2.2 MB/s 2024-04-25T01:20:46.573 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:20:46.629 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T01:20:46.629 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T01:20:46.629 INFO:tasks.ceph:Generating config... 2024-04-25T01:20:46.630 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-25T01:20:46.631 INFO:tasks.ceph:[client] debug client = 20 2024-04-25T01:20:46.631 INFO:tasks.ceph:[client] debug ms = 1 2024-04-25T01:20:46.631 INFO:tasks.ceph:[client] pid file = 2024-04-25T01:20:46.631 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-25T01:20:46.631 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-25T01:20:46.631 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-25T01:20:46.631 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-25T01:20:46.631 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-25T01:20:46.631 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-25T01:20:46.631 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-25T01:20:46.631 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-25T01:20:46.631 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-25T01:20:46.631 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-25T01:20:46.631 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-25T01:20:46.631 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-25T01:20:46.632 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-25T01:20:46.632 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-25T01:20:46.632 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T01:20:46.632 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T01:20:46.632 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T01:20:46.632 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T01:20:46.632 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T01:20:46.632 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-25T01:20:46.632 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-25T01:20:46.632 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T01:20:46.632 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T01:20:46.632 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-25T01:20:46.632 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-25T01:20:46.632 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T01:20:46.632 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T01:20:46.632 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-25T01:20:46.633 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T01:20:46.633 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T01:20:46.633 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T01:20:46.633 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T01:20:46.633 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T01:20:46.633 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-25T01:20:46.633 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T01:20:46.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T01:20:46.687 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T01:20:46.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T01:20:46.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T01:20:46.799 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.77'), ('mon.b', '172.21.15.121'), ('mon.c', '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]')] 2024-04-25T01:20:46.799 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.77,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-25T01:20:46.800 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:20:46.800 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T01:20:46.855 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.77 --add b 172.21.15.121 --addv c '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:20:46.941 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:20:46.941 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: generated fsid 7c3c39ee-077e-476c-8994-eede4fab301d 2024-04-25T01:20:46.941 INFO:teuthology.orchestra.run.smithi077.stdout:setting min_mon_release = quincy 2024-04-25T01:20:46.941 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 0 2024-04-25T01:20:46.941 INFO:teuthology.orchestra.run.smithi077.stdout:fsid 7c3c39ee-077e-476c-8994-eede4fab301d 2024-04-25T01:20:46.941 INFO:teuthology.orchestra.run.smithi077.stdout:last_changed 2024-04-25T01:20:46.960580+0000 2024-04-25T01:20:46.942 INFO:teuthology.orchestra.run.smithi077.stdout:created 2024-04-25T01:20:46.960580+0000 2024-04-25T01:20:46.942 INFO:teuthology.orchestra.run.smithi077.stdout:min_mon_release 17 (quincy) 2024-04-25T01:20:46.942 INFO:teuthology.orchestra.run.smithi077.stdout:election_strategy: 1 2024-04-25T01:20:46.942 INFO:teuthology.orchestra.run.smithi077.stdout:0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-04-25T01:20:46.942 INFO:teuthology.orchestra.run.smithi077.stdout:1: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.b 2024-04-25T01:20:46.942 INFO:teuthology.orchestra.run.smithi077.stdout:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-04-25T01:20:46.942 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-25T01:20:46.943 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T01:20:46.958 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7c3c39ee-077e-476c-8994-eede4fab301d... 2024-04-25T01:20:46.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T01:20:47.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T01:20:47.038 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-04-25T01:20:47.038 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-04-25T01:20:47.038 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T01:20:47.038 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-04-25T01:20:47.038 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.038 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-04-25T01:20:47.038 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.038 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-04-25T01:20:47.038 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.038 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-04-25T01:20:47.038 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-04-25T01:20:47.039 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-04-25T01:20:47.040 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.77,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on skipped message = False 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 7c3c39ee-077e-476c-8994-eede4fab301d 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-04-25T01:20:47.041 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-04-25T01:20:47.042 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 20 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 20 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 10 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.043 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-04-25T01:20:47.044 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.045 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: mon down mkfs grace = 300 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T01:20:47.046 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = "" 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2024-04-25T01:20:47.047 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds balancer = 20 2024-04-25T01:20:47.048 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-04-25T01:20:47.048 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2024-04-25T01:20:47.048 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2024-04-25T01:20:47.048 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2024-04-25T01:20:47.048 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2024-04-25T01:20:47.048 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-04-25T01:20:47.048 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-04-25T01:20:47.048 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-04-25T01:20:47.048 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-04-25T01:20:47.049 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-04-25T01:20:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.77,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-25T01:20:47.051 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on skipped message = False 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 7c3c39ee-077e-476c-8994-eede4fab301d 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-04-25T01:20:47.052 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-04-25T01:20:47.053 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 20 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 20 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 10 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-04-25T01:20:47.054 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-04-25T01:20:47.055 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-04-25T01:20:47.056 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: mon down mkfs grace = 300 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = "" 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-25T01:20:47.057 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds balancer = 20 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2024-04-25T01:20:47.058 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2024-04-25T01:20:47.059 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-25T01:20:47.059 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-25T01:20:47.059 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-25T01:20:47.060 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T01:20:47.060 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T01:20:47.152 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T01:20:47.152 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:20:47.152 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T01:20:47.166 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:20:47.167 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T01:20:47.222 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:20:47.222 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:20:47.222 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T01:20:47.222 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T01:20:47.294 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:20:47.294 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:20:47.349 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:20:47.349 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:20:47.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T01:20:47.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T01:20:47.383 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:20:47.383 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:20:47.439 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T01:20:47.439 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T01:20:47.439 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T01:20:47.486 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T01:20:47.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T01:20:47.543 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T01:20:47.546 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T01:20:47.546 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T01:20:47.594 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T01:20:47.597 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-25T01:20:47.620 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T01:20:47.707 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T01:20:47.710 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-25T01:20:47.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T01:20:47.782 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T01:20:47.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-25T01:20:47.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T01:20:47.895 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T01:20:47.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-25T01:20:47.920 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T01:20:47.920 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T01:20:47.959 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T01:20:47.971 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-25T01:20:48.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-25T01:20:48.101 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-25T01:20:48.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-25T01:20:48.192 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-25T01:20:48.204 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T01:20:48.204 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T01:20:48.204 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T01:20:48.228 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T01:20:48.228 INFO:tasks.ceph:role: osd.0 2024-04-25T01:20:48.228 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:20:48.228 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T01:20:48.299 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:20:48.299 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:20:48.299 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:20:48.299 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:20:48.299 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:20:48.299 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:20:48.299 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:20:48.299 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:20:48.299 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:20:48.299 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:20:48.815 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-04-25T01:20:48.817 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-04-25T01:20:48.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T01:20:48.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T01:20:48.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T01:20:49.029 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T01:20:49.029 INFO:tasks.ceph:role: osd.1 2024-04-25T01:20:49.029 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:20:49.030 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T01:20:49.102 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:20:49.102 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:20:49.102 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:20:49.102 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:20:49.102 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:20:49.102 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:20:49.102 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:20:49.102 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:20:49.103 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:20:49.103 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:20:49.558 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-04-25T01:20:49.561 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-04-25T01:20:49.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-25T01:20:49.654 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T01:20:49.681 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T01:20:49.744 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T01:20:49.744 INFO:tasks.ceph:role: osd.2 2024-04-25T01:20:49.745 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:20:49.745 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T01:20:49.815 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:20:49.815 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:20:49.815 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:20:49.815 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:20:49.815 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:20:49.815 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:20:49.815 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:20:49.815 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:20:49.815 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:20:49.815 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:20:50.304 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-04-25T01:20:50.306 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-04-25T01:20:50.306 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-25T01:20:50.443 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T01:20:50.471 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T01:20:50.534 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T01:20:50.534 INFO:tasks.ceph:role: osd.3 2024-04-25T01:20:50.534 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:20:50.534 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T01:20:50.604 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:20:50.604 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:20:50.605 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:20:50.605 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:20:50.605 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:20:50.605 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:20:50.605 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:20:50.605 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:20:50.605 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:20:50.605 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:20:51.121 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-04-25T01:20:51.123 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-04-25T01:20:51.123 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-25T01:20:51.247 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T01:20:51.277 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:20:51.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:51.397+0000 7fb17d40c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-25T01:20:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:51.397+0000 7fb17d40c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-25T01:20:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:51.397+0000 7fb17d40c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T01:20:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:51.397+0000 7fb17d40c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T01:20:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:51.397+0000 7fb17d40c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T01:20:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:51.398+0000 7fb17d40c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-25T01:20:54.236 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T01:20:54.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:20:54.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:54.368+0000 7f0d1abcb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-25T01:20:54.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:54.368+0000 7f0d1abcb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-25T01:20:54.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:54.368+0000 7f0d1abcb640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T01:20:54.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:54.368+0000 7f0d1abcb640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T01:20:54.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:54.368+0000 7f0d1abcb640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T01:20:54.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:54.369+0000 7f0d1abcb640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-25T01:20:57.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T01:20:57.226 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:20:57.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:57.333+0000 7f868dc2d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-25T01:20:57.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:57.334+0000 7f868dc2d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-25T01:20:57.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:57.334+0000 7f868dc2d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T01:20:57.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:57.334+0000 7f868dc2d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T01:20:57.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:57.334+0000 7f868dc2d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T01:20:57.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:20:57.334+0000 7f868dc2d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-25T01:21:00.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T01:21:00.191 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:21:00.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:00.298+0000 7fa34821e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-25T01:21:00.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:00.299+0000 7fa34821e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-25T01:21:00.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:00.299+0000 7fa34821e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T01:21:00.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:00.299+0000 7fa34821e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T01:21:00.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:00.299+0000 7fa34821e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T01:21:00.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:00.299+0000 7fa34821e640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-25T01:21:03.127 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T01:21:03.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-25T01:21:03.180 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T01:21:03.180 INFO:tasks.ceph:role: osd.4 2024-04-25T01:21:03.180 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:21:03.180 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T01:21:03.254 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:21:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:21:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:21:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:21:03.255 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:21:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:21:03.255 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:21:03.255 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:21:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:21:03.255 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:21:03.760 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-25T01:21:03.762 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-25T01:21:03.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-25T01:21:03.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-25T01:21:03.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-25T01:21:03.995 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T01:21:03.995 INFO:tasks.ceph:role: osd.5 2024-04-25T01:21:03.995 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:21:03.995 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T01:21:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:21:04.066 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:21:04.066 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:21:04.066 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:21:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:21:04.066 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:21:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:21:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:21:04.066 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:21:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:21:04.525 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-25T01:21:04.527 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-25T01:21:04.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-25T01:21:04.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-25T01:21:04.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-25T01:21:04.707 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T01:21:04.709 INFO:tasks.ceph:role: osd.6 2024-04-25T01:21:04.710 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:21:04.710 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T01:21:04.778 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:21:04.778 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:21:04.778 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:21:04.779 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:21:04.779 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:21:04.779 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:21:04.779 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:21:04.779 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:21:04.779 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:21:04.779 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:21:05.272 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-25T01:21:05.274 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-25T01:21:05.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-25T01:21:05.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-25T01:21:05.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-25T01:21:05.523 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T01:21:05.523 INFO:tasks.ceph:role: osd.7 2024-04-25T01:21:05.523 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:21:05.523 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T01:21:05.593 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:21:05.594 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:21:05.594 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:21:05.594 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:21:05.594 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:21:05.594 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:21:05.594 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:21:05.594 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:21:05.594 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:21:05.594 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:21:06.120 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-25T01:21:06.122 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-25T01:21:06.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-25T01:21:06.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-25T01:21:06.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:21:06.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:06.416+0000 7f8d56a1c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-25T01:21:06.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:06.417+0000 7f8d56a1c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-25T01:21:06.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:06.417+0000 7f8d56a1c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T01:21:06.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:06.417+0000 7f8d56a1c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T01:21:06.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:06.417+0000 7f8d56a1c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T01:21:06.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:06.417+0000 7f8d56a1c640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-25T01:21:09.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-25T01:21:09.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:21:09.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:09.378+0000 7f906ac30640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-25T01:21:09.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:09.378+0000 7f906ac30640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-25T01:21:09.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:09.378+0000 7f906ac30640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T01:21:09.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:09.378+0000 7f906ac30640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T01:21:09.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:09.378+0000 7f906ac30640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T01:21:09.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:09.379+0000 7f906ac30640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-25T01:21:12.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-25T01:21:12.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:21:12.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:12.340+0000 7f770b957640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-25T01:21:12.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:12.340+0000 7f770b957640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-25T01:21:12.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:12.340+0000 7f770b957640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T01:21:12.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:12.340+0000 7f770b957640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T01:21:12.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:12.340+0000 7f770b957640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T01:21:12.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:12.341+0000 7f770b957640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-25T01:21:15.086 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:21:15.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:21:15.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-25T01:21:15.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:21:15.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:15.342+0000 7f9106230640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-25T01:21:15.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:15.342+0000 7f9106230640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-25T01:21:15.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:15.342+0000 7f9106230640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T01:21:15.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:15.342+0000 7f9106230640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T01:21:15.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:15.342+0000 7f9106230640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T01:21:15.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:15.343+0000 7f9106230640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-25T01:21:18.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-25T01:21:18.244 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T01:21:18.244 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:18.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-25T01:21:18.272 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:18.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-25T01:21:18.339 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:18.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-25T01:21:18.405 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:18.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T01:21:18.471 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:18.471 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T01:21:18.535 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:18.536 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T01:21:18.599 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:18.599 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T01:21:18.664 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:18.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T01:21:18.692 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:18.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-25T01:21:18.760 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:18.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-25T01:21:18.825 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:18.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-25T01:21:18.891 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:18.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-25T01:21:18.956 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:18.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-25T01:21:19.021 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:19.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-25T01:21:19.086 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:19.086 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T01:21:19.103 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:19.103 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-25T01:21:19.159 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:19.159 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-25T01:21:19.175 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:19.175 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-25T01:21:19.231 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T01:21:19.231 DEBUG:teuthology.orchestra.run.smithi077:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T01:21:19.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T01:21:19.275 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.y] 2024-04-25T01:21:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvrylmqtQqHhAAaEIOEzQbQjkAPt2EIRAixQ== 2024-04-25T01:21:19.275 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-04-25T01:21:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvrylmRBubJBAA1jgiWa9DTdamG6ZHPTzJHA== 2024-04-25T01:21:19.275 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-04-25T01:21:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvrylmc1hYKxAATtGeOU4Ep9JGOclelSbShw== 2024-04-25T01:21:19.275 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2024-04-25T01:21:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDzrylmFiLEFxAAiu3CMixQObEnTzVB4+oIOg== 2024-04-25T01:21:19.275 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2024-04-25T01:21:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD2rylm0cMJFhAAVluT5EojBn9sI3GaJjJRqw== 2024-04-25T01:21:19.275 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD5rylmklH8ExAASXBlrt1JxxxhCynyR08Isw== 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.3] 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD8rylmKQ3jERAARBac0vky2rIDzUdr4nYn/A== 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.x] 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvrylmw1qCIRAAqlQEapgQsaTD1GhzKlJX/A== 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvrylmxEjKLxAAaPXHEf3AJlrRvnyH4OKCpg== 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvrylm8tWDNhAAq3irRf+ZuOWsrOLP59WOuA== 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.4] 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQACsClmaGfoGBAAtbFXxGHH/zXHHDB3UBzlTA== 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.5] 2024-04-25T01:21:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAFsClm7GCeFhAANCOnPFNJhLmUfK+Kk+zHUw== 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.6] 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAIsClmC2hZFBAAEem007DX6DrvGHwN5YTjgw== 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.7] 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQALsClmijV3FBAANujp5DQhJDc8IYocyllBBA== 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvrylmsfhcOhAAYZeVeMoyKmZdON0jUeFmeQ== 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout:[client.2] 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDwrylmRdgcBBAA4/WiRMEX2/s9wbevmTPgLA== 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout:[client.1] 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDwrylm8sQzBxAAJNZ00VUA4tzDCB+UZb+gnQ== 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout:[client.3] 2024-04-25T01:21:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDwrylmG5KlDBAA28sSTbOA+nZ1ejbMP6Mqig== 2024-04-25T01:21:19.294 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.y] 2024-04-25T01:21:19.294 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDvrylmqtQqHhAAaEIOEzQbQjkAPt2EIRAixQ== 2024-04-25T01:21:19.294 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2024-04-25T01:21:19.294 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDvrylmRBubJBAA1jgiWa9DTdamG6ZHPTzJHA== 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDvrylmc1hYKxAATtGeOU4Ep9JGOclelSbShw== 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDzrylmFiLEFxAAiu3CMixQObEnTzVB4+oIOg== 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQD2rylm0cMJFhAAVluT5EojBn9sI3GaJjJRqw== 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQD5rylmklH8ExAASXBlrt1JxxxhCynyR08Isw== 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQD8rylmKQ3jERAARBac0vky2rIDzUdr4nYn/A== 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDvrylmw1qCIRAAqlQEapgQsaTD1GhzKlJX/A== 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2024-04-25T01:21:19.295 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDvrylmxEjKLxAAaPXHEf3AJlrRvnyH4OKCpg== 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDvrylm8tWDNhAAq3irRf+ZuOWsrOLP59WOuA== 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.4] 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQACsClmaGfoGBAAtbFXxGHH/zXHHDB3UBzlTA== 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.5] 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAFsClm7GCeFhAANCOnPFNJhLmUfK+Kk+zHUw== 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.6] 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAIsClmC2hZFBAAEem007DX6DrvGHwN5YTjgw== 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.7] 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQALsClmijV3FBAANujp5DQhJDc8IYocyllBBA== 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDvrylmsfhcOhAAYZeVeMoyKmZdON0jUeFmeQ== 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout:[client.2] 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDwrylmRdgcBBAA4/WiRMEX2/s9wbevmTPgLA== 2024-04-25T01:21:19.296 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2024-04-25T01:21:19.297 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDwrylm8sQzBxAAJNZ00VUA4tzDCB+UZb+gnQ== 2024-04-25T01:21:19.297 INFO:teuthology.orchestra.run.smithi121.stdout:[client.3] 2024-04-25T01:21:19.297 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDwrylmG5KlDBAA28sSTbOA+nZ1ejbMP6Mqig== 2024-04-25T01:21:19.297 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T01:21:19.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T01:21:19.386 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:21:19.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:21:19.440 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:21:19.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:21:19.493 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.593 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.643 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.694 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T01:21:19.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T01:21:19.745 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:21:19.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:21:19.798 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:21:19.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:21:19.854 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.905 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.955 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:19.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:20.005 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:20.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:21:20.055 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:21:20.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:21:20.108 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:21:20.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:21:20.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:21:20.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:21:20.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:21:20.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:21:20.268 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T01:21:20.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T01:21:20.292 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T01:21:20.384 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T01:21:20.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-25T01:21:20.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T01:21:20.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-25T01:21:20.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-25T01:21:20.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T01:21:20.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-25T01:21:20.726 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:21:20.728 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:21:20.780 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T01:21:20.780 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T01:21:20.780 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T01:21:20.783 INFO:tasks.ceph.mon.a:Started 2024-04-25T01:21:20.783 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T01:21:20.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T01:21:20.822 INFO:tasks.ceph.mon.b:Started 2024-04-25T01:21:20.823 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-25T01:21:20.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-25T01:21:20.825 INFO:tasks.ceph.mon.c:Started 2024-04-25T01:21:20.825 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T01:21:20.825 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-25T01:21:20.825 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-25T01:21:20.827 INFO:tasks.ceph.mgr.y:Started 2024-04-25T01:21:20.828 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T01:21:20.828 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T01:21:20.830 INFO:tasks.ceph.mgr.x:Started 2024-04-25T01:21:20.830 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T01:21:20.830 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T01:21:20.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.946+0000 7eff1b038640 1 Processor -- start 2024-04-25T01:21:20.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.946+0000 7eff1b038640 1 -- start start 2024-04-25T01:21:20.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff1b038640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 0x7eff140b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:20.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff1b038640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7eff1406c610 0x7eff1406cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:20.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff1b038640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff140d0a90 con 0x7eff14161df0 2024-04-25T01:21:20.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff1b038640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7eff14162d00 con 0x7eff1406c610 2024-04-25T01:21:20.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff1b038640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7eff140d0d70 con 0x7eff140b2920 2024-04-25T01:21:20.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff13fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 0x7eff140b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:20.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff195d5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7eff1406c610 0x7eff1406cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff13fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 0x7eff140b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:58786/0 (socket says 172.21.15.77:58786) 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff195d5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7eff1406c610 0x7eff1406cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:53412/0 (socket says 172.21.15.77:53412) 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff13fff640 1 -- 172.21.15.77:0/2862814736 learned_addr learned my addr 172.21.15.77:0/2862814736 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff18dd4640 1 --1- 172.21.15.77:0/2862814736 >> v1:172.21.15.121:6789/0 conn(0x7eff14161df0 0x7eff141621d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:37814/0 (socket says 172.21.15.77:37814) 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff195d5640 1 -- 172.21.15.77:0/2862814736 >> v2:172.21.15.121:3300/0 conn(0x7eff1406c610 msgr2=0x7eff1406cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff195d5640 1 -- 172.21.15.77:0/2862814736 >> v2:172.21.15.121:3300/0 conn(0x7eff1406c610 msgr2=0x7eff1406cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff195d5640 1 --2- 172.21.15.77:0/2862814736 >> v2:172.21.15.121:3300/0 conn(0x7eff1406c610 0x7eff1406cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff195d5640 1 --2- 172.21.15.77:0/2862814736 >> v2:172.21.15.121:3300/0 conn(0x7eff1406c610 0x7eff1406cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff13fff640 1 -- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 msgr2=0x7eff140b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.947+0000 7eff13fff640 1 -- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 msgr2=0x7eff140b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.948+0000 7eff13fff640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 0x7eff140b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:21:20.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:20.948+0000 7eff13fff640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 0x7eff140b42c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:21:21.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.115+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 345226359 0 0) 0x7eff140d0a90 con 0x7eff14161df0 2024-04-25T01:21:21.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.115+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff04003520 con 0x7eff14161df0 2024-04-25T01:21:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 915466798 0 0) 0x7eff04003520 con 0x7eff14161df0 2024-04-25T01:21:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 msgr2=0x7eff140b42c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:21:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 0x7eff140b42c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 >> v2:172.21.15.121:3300/0 conn(0x7eff1406c610 msgr2=0x7eff1406cac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:21:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 --2- 172.21.15.77:0/2862814736 >> v2:172.21.15.121:3300/0 conn(0x7eff1406c610 0x7eff1406cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff140d1050 con 0x7eff14161df0 2024-04-25T01:21:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7eff08001670 con 0x7eff14161df0 2024-04-25T01:21:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 >> v1:172.21.15.121:6789/0 conn(0x7eff14161df0 legacy=0x7eff141621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff040039b0 0x7eff04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff04004800 0x7eff04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 0x7eff04005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff14162d00 con 0x7eff04004800 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7eff140d0d70 con 0x7eff040039b0 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7eff0400bc00 con 0x7eff140b2920 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff13fff640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff04004800 0x7eff04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff18dd4640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff040039b0 0x7eff04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff195d5640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 0x7eff04005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff13fff640 1 -- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 msgr2=0x7eff04005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff13fff640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff140b2920 0x7eff04005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff13fff640 1 -- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff040039b0 msgr2=0x7eff04003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff13fff640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff040039b0 0x7eff04003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.122+0000 7eff13fff640 1 -- 172.21.15.77:0/2862814736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff04003520 con 0x7eff04004800 2024-04-25T01:21:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.123+0000 7eff13fff640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff04004800 0x7eff04004c80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff00009100 tx=0x7eff04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60cafd0bda2cba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.123+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff00037070 con 0x7eff04004800 2024-04-25T01:21:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.123+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff0002ad00 con 0x7eff04004800 2024-04-25T01:21:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.123+0000 7eff137fe640 1 -- 172.21.15.77:0/2862814736 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0000f040 con 0x7eff04004800 2024-04-25T01:21:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.123+0000 7eff1b038640 1 -- 172.21.15.77:0/2862814736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff04004800 msgr2=0x7eff04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.123+0000 7eff1b038640 1 --2- 172.21.15.77:0/2862814736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff04004800 0x7eff04004c80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff00009100 tx=0x7eff04005830 comp rx=0 tx=0).stop 2024-04-25T01:21:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.124+0000 7eff195d5640 1 -- 172.21.15.77:0/2862814736 reap_dead start 2024-04-25T01:21:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.124+0000 7eff1b038640 1 -- 172.21.15.77:0/2862814736 shutdown_connections 2024-04-25T01:21:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.124+0000 7eff1b038640 1 -- 172.21.15.77:0/2862814736 >> 172.21.15.77:0/2862814736 conn(0x7eff140b37a0 msgr2=0x7eff140b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.124+0000 7eff1b038640 1 -- 172.21.15.77:0/2862814736 shutdown_connections 2024-04-25T01:21:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.124+0000 7eff1b038640 1 -- 172.21.15.77:0/2862814736 wait complete. 2024-04-25T01:21:21.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.125+0000 7eff1b038640 1 Processor -- start 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.125+0000 7eff1b038640 1 -- start start 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff1b038640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff04004800 0x7eff141550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff1b038640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff140c78f0 0x7eff141548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff1b038640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff1406eea0 0x7eff1406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff1b038640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff14162a60 con 0x7eff140c78f0 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff1b038640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7eff141628e0 con 0x7eff1406eea0 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff1b038640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7eff140aea80 con 0x7eff04004800 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff140c78f0 0x7eff141548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff18dd4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff04004800 0x7eff141550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff140c78f0 0x7eff141548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45868/0 (socket says 172.21.15.77:45868) 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff13fff640 1 -- 172.21.15.77:0/3534520050 learned_addr learned my addr 172.21.15.77:0/3534520050 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff195d5640 1 --2- 172.21.15.77:0/3534520050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff1406eea0 0x7eff1406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff13fff640 1 -- 172.21.15.77:0/3534520050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff04004800 msgr2=0x7eff141550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff13fff640 1 --2- 172.21.15.77:0/3534520050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff04004800 0x7eff141550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff13fff640 1 -- 172.21.15.77:0/3534520050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff1406eea0 msgr2=0x7eff1406f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff13fff640 1 --2- 172.21.15.77:0/3534520050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff1406eea0 0x7eff1406f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff13fff640 1 -- 172.21.15.77:0/3534520050 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff140cc2a0 con 0x7eff140c78f0 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff13fff640 1 --2- 172.21.15.77:0/3534520050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff140c78f0 0x7eff141548a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7eff00035040 tx=0x7eff00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff117fa640 1 -- 172.21.15.77:0/3534520050 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff00042020 con 0x7eff140c78f0 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff117fa640 1 -- 172.21.15.77:0/3534520050 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff0000f000 con 0x7eff140c78f0 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff1b038640 1 -- 172.21.15.77:0/3534520050 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff140cb790 con 0x7eff140c78f0 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff117fa640 1 -- 172.21.15.77:0/3534520050 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff00037040 con 0x7eff140c78f0 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.126+0000 7eff1b038640 1 -- 172.21.15.77:0/3534520050 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff140cc560 con 0x7eff140c78f0 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.127+0000 7eff117fa640 1 -- 172.21.15.77:0/3534520050 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7eff00045020 con 0x7eff140c78f0 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.127+0000 7eff117fa640 1 -- 172.21.15.77:0/3534520050 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7eff0003e7e0 con 0x7eff140c78f0 2024-04-25T01:21:21.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.127+0000 7eff1b038640 1 -- 172.21.15.77:0/3534520050 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff14068180 con 0x7eff140c78f0 2024-04-25T01:21:21.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.129+0000 7eff117fa640 1 -- 172.21.15.77:0/3534520050 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7eff0004c030 con 0x7eff140c78f0 2024-04-25T01:21:21.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.195+0000 7eff1b038640 1 -- 172.21.15.77:0/3534520050 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7eff140b7bf0 con 0x7eff140c78f0 2024-04-25T01:21:21.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.197+0000 7eff117fa640 1 -- 172.21.15.77:0/3534520050 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7eff0003dc00 con 0x7eff140c78f0 2024-04-25T01:21:21.178 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2024-04-25T01:21:21.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.198+0000 7eff1b038640 1 -- 172.21.15.77:0/3534520050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff140c78f0 msgr2=0x7eff141548a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.198+0000 7eff1b038640 1 --2- 172.21.15.77:0/3534520050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff140c78f0 0x7eff141548a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7eff00035040 tx=0x7eff00004730 comp rx=0 tx=0).stop 2024-04-25T01:21:21.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.198+0000 7eff1b038640 1 -- 172.21.15.77:0/3534520050 shutdown_connections 2024-04-25T01:21:21.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.198+0000 7eff1b038640 1 --2- 172.21.15.77:0/3534520050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff140c78f0 0x7eff141548a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.198+0000 7eff1b038640 1 --2- 172.21.15.77:0/3534520050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff1406eea0 0x7eff1406f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.198+0000 7eff1b038640 1 --2- 172.21.15.77:0/3534520050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff04004800 0x7eff141550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.198+0000 7eff1b038640 1 -- 172.21.15.77:0/3534520050 >> 172.21.15.77:0/3534520050 conn(0x7eff140b37a0 msgr2=0x7eff140c7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:21.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.198+0000 7eff1b038640 1 -- 172.21.15.77:0/3534520050 shutdown_connections 2024-04-25T01:21:21.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:21.198+0000 7eff1b038640 1 -- 172.21.15.77:0/3534520050 wait complete. 2024-04-25T01:21:21.190 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T01:21:21.190 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T01:21:21.190 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:21.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T01:21:21.203 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:21.222+0000 7fad55410040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:21:21.204 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:21.222+0000 7f4756320040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:21:21.213 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:21.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T01:21:21.278 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:21.278 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T01:21:21.341 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:21.341 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T01:21:21.346 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:21.365+0000 7fad55410040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:21:21.360 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:21.379+0000 7f4756320040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:21:21.383 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:21.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-25T01:21:21.400 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:21.419+0000 7fad55410040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:21:21.406 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:21.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-25T01:21:21.415 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:21.434+0000 7f4756320040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:21:21.470 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:21.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-25T01:21:21.534 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:21:21.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-25T01:21:21.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 40e3ffde-0d89-45b6-acf6-b17d1c5a5114 0 2024-04-25T01:21:21.641 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:21.659+0000 7fad55410040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:21:21.657 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:21.676+0000 7f4756320040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:21:21.689 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:21.708+0000 7fad55410040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:21:21.702 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:21.721+0000 7f4756320040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:21:21.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.754+0000 7f3dbe55a640 1 Processor -- start 2024-04-25T01:21:21.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.754+0000 7f3dbe55a640 1 -- start start 2024-04-25T01:21:21.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.755+0000 7f3dbe55a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3db80d2780 0x7f3db80cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.755+0000 7f3dbe55a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3db80cfa80 con 0x7f3db80d0600 2024-04-25T01:21:21.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.755+0000 7f3dbe55a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3db80cf1e0 con 0x7f3db815f720 2024-04-25T01:21:21.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.755+0000 7f3dbe55a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3db80cf830 con 0x7f3db80d2780 2024-04-25T01:21:21.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.755+0000 7f3dbcaf7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3db80d2780 0x7f3db80cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.755+0000 7f3dbcaf7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3db80d2780 0x7f3db80cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39202/0 (socket says 172.21.15.121:39202) 2024-04-25T01:21:21.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.755+0000 7f3db7fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f3db815f720 0x7f3db80cfef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:42976/0 (socket says 172.21.15.121:42976) 2024-04-25T01:21:21.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.755+0000 7f3dbcaf7640 1 -- 172.21.15.121:0/3768857841 learned_addr learned my addr 172.21.15.121:0/3768857841 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:21.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.755+0000 7f3dbcaf7640 1 -- 172.21.15.121:0/3768857841 >> v1:172.21.15.121:6789/0 conn(0x7f3db815f720 legacy=0x7f3db80cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.755+0000 7f3dbcaf7640 1 -- 172.21.15.121:0/3768857841 >> v1:172.21.15.77:6789/0 conn(0x7f3db80d0600 legacy=0x7f3db80ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.755+0000 7f3dbcaf7640 1 -- 172.21.15.121:0/3768857841 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db80b15c0 con 0x7f3db80d2780 2024-04-25T01:21:21.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.756+0000 7f3dbcaf7640 1 --2- 172.21.15.121:0/3768857841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db80d2780 0x7f3db80cd260 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3db0009e50 tx=0x7f3db002a7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fdb9f8958203f0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:21.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.756+0000 7f3db6ffd640 1 -- 172.21.15.121:0/3768857841 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0036070 con 0x7f3db80d2780 2024-04-25T01:21:21.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.756+0000 7f3db6ffd640 1 -- 172.21.15.121:0/3768857841 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db0031600 con 0x7f3db80d2780 2024-04-25T01:21:21.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.756+0000 7f3db6ffd640 1 -- 172.21.15.121:0/3768857841 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db00319f0 con 0x7f3db80d2780 2024-04-25T01:21:21.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.757+0000 7f3dbe55a640 1 -- 172.21.15.121:0/3768857841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db80d2780 msgr2=0x7f3db80cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.757+0000 7f3dbe55a640 1 --2- 172.21.15.121:0/3768857841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db80d2780 0x7f3db80cd260 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3db0009e50 tx=0x7f3db002a7c0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.757+0000 7f3dbe55a640 1 -- 172.21.15.121:0/3768857841 shutdown_connections 2024-04-25T01:21:21.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.757+0000 7f3dbe55a640 1 --2- 172.21.15.121:0/3768857841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db80d2780 0x7f3db80cd260 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.757+0000 7f3dbe55a640 1 -- 172.21.15.121:0/3768857841 >> 172.21.15.121:0/3768857841 conn(0x7f3db80d1c60 msgr2=0x7f3db80d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:21.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.757+0000 7f3dbe55a640 1 -- 172.21.15.121:0/3768857841 shutdown_connections 2024-04-25T01:21:21.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.757+0000 7f3dbe55a640 1 -- 172.21.15.121:0/3768857841 wait complete. 2024-04-25T01:21:21.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.758+0000 7f3dbe55a640 1 Processor -- start 2024-04-25T01:21:21.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.758+0000 7f3dbe55a640 1 -- start start 2024-04-25T01:21:21.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbe55a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3db80d0600 0x7f3db8150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbe55a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db80d2780 0x7f3db80ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbe55a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db815f720 0x7f3db80afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbe55a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3db80b1350 con 0x7f3db80d0600 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbe55a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3db80b11d0 con 0x7f3db80d2780 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbe55a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3db80b0b30 con 0x7f3db815f720 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3db77fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db80d2780 0x7f3db80ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbcaf7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db815f720 0x7f3db80afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbcaf7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db815f720 0x7f3db80afc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36984/0 (socket says 172.21.15.121:36984) 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbcaf7640 1 -- 172.21.15.121:0/405484532 learned_addr learned my addr 172.21.15.121:0/405484532 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3db77fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db80d2780 0x7f3db80ca480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39206/0 (socket says 172.21.15.121:39206) 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3db7fff640 1 --2- 172.21.15.121:0/405484532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3db80d0600 0x7f3db8150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbcaf7640 1 -- 172.21.15.121:0/405484532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db80d2780 msgr2=0x7f3db80ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbcaf7640 1 --2- 172.21.15.121:0/405484532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db80d2780 0x7f3db80ca480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbcaf7640 1 -- 172.21.15.121:0/405484532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3db80d0600 msgr2=0x7f3db8150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbcaf7640 1 --2- 172.21.15.121:0/405484532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3db80d0600 0x7f3db8150300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbcaf7640 1 -- 172.21.15.121:0/405484532 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db80c7a40 con 0x7f3db815f720 2024-04-25T01:21:21.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3db77fe640 1 --2- 172.21.15.121:0/405484532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db80d2780 0x7f3db80ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:21.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.759+0000 7f3dbcaf7640 1 --2- 172.21.15.121:0/405484532 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db815f720 0x7f3db80afc40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3db0034040 tx=0x7f3db0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:21.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.760+0000 7f3db4ff9640 1 -- 172.21.15.121:0/405484532 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0036050 con 0x7f3db815f720 2024-04-25T01:21:21.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.760+0000 7f3db4ff9640 1 -- 172.21.15.121:0/405484532 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db0042070 con 0x7f3db815f720 2024-04-25T01:21:21.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.760+0000 7f3dbe55a640 1 -- 172.21.15.121:0/405484532 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db81f2d80 con 0x7f3db815f720 2024-04-25T01:21:21.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.760+0000 7f3db4ff9640 1 -- 172.21.15.121:0/405484532 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0009510 con 0x7f3db815f720 2024-04-25T01:21:21.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.760+0000 7f3dbe55a640 1 -- 172.21.15.121:0/405484532 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db80c5830 con 0x7f3db815f720 2024-04-25T01:21:21.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.760+0000 7f3db4ff9640 1 -- 172.21.15.121:0/405484532 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3db0004450 con 0x7f3db815f720 2024-04-25T01:21:21.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.760+0000 7f3db4ff9640 1 -- 172.21.15.121:0/405484532 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f3db0047aa0 con 0x7f3db815f720 2024-04-25T01:21:21.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.761+0000 7f3dbe55a640 1 -- 172.21.15.121:0/405484532 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3db8068180 con 0x7f3db815f720 2024-04-25T01:21:21.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.763+0000 7f3db4ff9640 1 -- 172.21.15.121:0/405484532 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3db0047460 con 0x7f3db815f720 2024-04-25T01:21:21.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.835+0000 7f3dbe55a640 1 -- 172.21.15.121:0/405484532 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "40e3ffde-0d89-45b6-acf6-b17d1c5a5114", "id": 0} v 0) -- 0x7f3db8073100 con 0x7f3db815f720 2024-04-25T01:21:21.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.841+0000 7f3db4ff9640 1 -- 172.21.15.121:0/405484532 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "40e3ffde-0d89-45b6-acf6-b17d1c5a5114", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f3db0047640 con 0x7f3db815f720 2024-04-25T01:21:21.823 INFO:teuthology.orchestra.run.smithi121.stdout:0 2024-04-25T01:21:21.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.842+0000 7f3dbe55a640 1 -- 172.21.15.121:0/405484532 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db815f720 msgr2=0x7f3db80afc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.842+0000 7f3dbe55a640 1 --2- 172.21.15.121:0/405484532 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db815f720 0x7f3db80afc40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3db0034040 tx=0x7f3db0004a40 comp rx=0 tx=0).stop 2024-04-25T01:21:21.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.842+0000 7f3dbe55a640 1 -- 172.21.15.121:0/405484532 shutdown_connections 2024-04-25T01:21:21.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.842+0000 7f3dbe55a640 1 --2- 172.21.15.121:0/405484532 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3db815f720 0x7f3db80afc40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.842+0000 7f3dbe55a640 1 --2- 172.21.15.121:0/405484532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db80d2780 0x7f3db80ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.842+0000 7f3dbe55a640 1 --2- 172.21.15.121:0/405484532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3db80d0600 0x7f3db8150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.842+0000 7f3dbe55a640 1 -- 172.21.15.121:0/405484532 >> 172.21.15.121:0/405484532 conn(0x7f3db80d1c60 msgr2=0x7f3db80b17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:21.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.842+0000 7f3dbe55a640 1 -- 172.21.15.121:0/405484532 shutdown_connections 2024-04-25T01:21:21.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.842+0000 7f3dbe55a640 1 -- 172.21.15.121:0/405484532 wait complete. 2024-04-25T01:21:21.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 3054f2ce-6486-4603-bc78-52bcefbd2aef 1 2024-04-25T01:21:21.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.990+0000 7f7e78f21640 1 Processor -- start 2024-04-25T01:21:21.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.990+0000 7f7e78f21640 1 -- start start 2024-04-25T01:21:21.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e78f21640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7e740cf370 0x7f7e740cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e78f21640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e7406dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e78f21640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e740cb900 con 0x7f7e740ccc00 2024-04-25T01:21:21.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e78f21640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f7e74072c70 con 0x7f7e740cf370 2024-04-25T01:21:21.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e78f21640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7e740cbba0 con 0x7f7e740cd060 2024-04-25T01:21:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e72d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e7406dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e72d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e7406dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36996/0 (socket says 172.21.15.121:36996) 2024-04-25T01:21:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e72d9d640 1 -- 172.21.15.121:0/3672711101 learned_addr learned my addr 172.21.15.121:0/3672711101 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e7259c640 1 --2- 172.21.15.121:0/3672711101 >> v2:172.21.15.77:3300/0 conn(0x7f7e740cf370 0x7f7e740cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e72d9d640 1 -- 172.21.15.121:0/3672711101 >> v2:172.21.15.77:3300/0 conn(0x7f7e740cf370 msgr2=0x7f7e740cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e72d9d640 1 --2- 172.21.15.121:0/3672711101 >> v2:172.21.15.77:3300/0 conn(0x7f7e740cf370 0x7f7e740cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e72d9d640 1 -- 172.21.15.121:0/3672711101 >> v1:172.21.15.77:6789/0 conn(0x7f7e740ccc00 legacy=0x7f7e74144210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.991+0000 7f7e72d9d640 1 -- 172.21.15.121:0/3672711101 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e740cfc90 con 0x7f7e740cd060 2024-04-25T01:21:21.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.992+0000 7f7e72d9d640 1 --2- 172.21.15.121:0/3672711101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e7406dd60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009e50 tx=0x7f7e6c02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81a51a9d765a58e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:21.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.992+0000 7f7e7159a640 1 -- 172.21.15.121:0/3672711101 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c036070 con 0x7f7e740cd060 2024-04-25T01:21:21.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.992+0000 7f7e7159a640 1 -- 172.21.15.121:0/3672711101 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e6c031600 con 0x7f7e740cd060 2024-04-25T01:21:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.992+0000 7f7e7159a640 1 -- 172.21.15.121:0/3672711101 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c0319d0 con 0x7f7e740cd060 2024-04-25T01:21:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.993+0000 7f7e78f21640 1 -- 172.21.15.121:0/3672711101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 msgr2=0x7f7e7406dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.993+0000 7f7e78f21640 1 --2- 172.21.15.121:0/3672711101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e7406dd60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009e50 tx=0x7f7e6c02a710 comp rx=0 tx=0).stop 2024-04-25T01:21:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.993+0000 7f7e78f21640 1 -- 172.21.15.121:0/3672711101 shutdown_connections 2024-04-25T01:21:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.993+0000 7f7e78f21640 1 --2- 172.21.15.121:0/3672711101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e7406dd60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.993+0000 7f7e78f21640 1 --2- 172.21.15.121:0/3672711101 >> v2:172.21.15.77:3300/0 conn(0x7f7e740cf370 0x7f7e740cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.993+0000 7f7e78f21640 1 -- 172.21.15.121:0/3672711101 >> 172.21.15.121:0/3672711101 conn(0x7f7e740d0bf0 msgr2=0x7f7e740d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.993+0000 7f7e78f21640 1 -- 172.21.15.121:0/3672711101 shutdown_connections 2024-04-25T01:21:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.993+0000 7f7e78f21640 1 -- 172.21.15.121:0/3672711101 wait complete. 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.994+0000 7f7e78f21640 1 Processor -- start 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.994+0000 7f7e78f21640 1 -- start start 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e78f21640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e740ccc00 0x7f7e740b5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e78f21640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e74156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e78f21640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e740cf370 0x7f7e74157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e78f21640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740cc400 con 0x7f7e740cf370 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e78f21640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740cc120 con 0x7f7e740ccc00 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e78f21640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7e740cc6e0 con 0x7f7e740cd060 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e71d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e74156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e7259c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e740ccc00 0x7f7e740b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e7259c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e740ccc00 0x7f7e740b5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39216/0 (socket says 172.21.15.121:39216) 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e7259c640 1 -- 172.21.15.121:0/3721839037 learned_addr learned my addr 172.21.15.121:0/3721839037 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e71d9b640 1 --2- 172.21.15.121:0/3721839037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e74156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37010/0 (socket says 172.21.15.121:37010) 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e72d9d640 1 --2- 172.21.15.121:0/3721839037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e740cf370 0x7f7e74157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e71d9b640 1 -- 172.21.15.121:0/3721839037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e740ccc00 msgr2=0x7f7e740b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e71d9b640 1 --2- 172.21.15.121:0/3721839037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e740ccc00 0x7f7e740b5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e71d9b640 1 -- 172.21.15.121:0/3721839037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e740cf370 msgr2=0x7f7e74157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e71d9b640 1 --2- 172.21.15.121:0/3721839037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e740cf370 0x7f7e74157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e71d9b640 1 -- 172.21.15.121:0/3721839037 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e74157b90 con 0x7f7e740cd060 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.995+0000 7f7e71d9b640 1 --2- 172.21.15.121:0/3721839037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e74156fd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800c3c0 tx=0x7f7e6800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.996+0000 7f7e62ffd640 1 -- 172.21.15.121:0/3721839037 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680052d0 con 0x7f7e740cd060 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.996+0000 7f7e62ffd640 1 -- 172.21.15.121:0/3721839037 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e68005490 con 0x7f7e740cd060 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.996+0000 7f7e78f21640 1 -- 172.21.15.121:0/3721839037 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e74157e70 con 0x7f7e740cd060 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.996+0000 7f7e62ffd640 1 -- 172.21.15.121:0/3721839037 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68007260 con 0x7f7e740cd060 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.996+0000 7f7e78f21640 1 -- 172.21.15.121:0/3721839037 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e74158430 con 0x7f7e740cd060 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.996+0000 7f7e62ffd640 1 -- 172.21.15.121:0/3721839037 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7e68007400 con 0x7f7e740cd060 2024-04-25T01:21:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.996+0000 7f7e62ffd640 1 -- 172.21.15.121:0/3721839037 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f7e68003b20 con 0x7f7e740cd060 2024-04-25T01:21:21.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:21.997+0000 7f7e78f21640 1 -- 172.21.15.121:0/3721839037 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e74068180 con 0x7f7e740cd060 2024-04-25T01:21:21.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.000+0000 7f7e62ffd640 1 -- 172.21.15.121:0/3721839037 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7e68003d70 con 0x7f7e740cd060 2024-04-25T01:21:22.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.073+0000 7f7e78f21640 1 -- 172.21.15.121:0/3721839037 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3054f2ce-6486-4603-bc78-52bcefbd2aef", "id": 1} v 0) -- 0x7f7e7406f8b0 con 0x7f7e740cd060 2024-04-25T01:21:22.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.079+0000 7f7e62ffd640 1 -- 172.21.15.121:0/3721839037 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3054f2ce-6486-4603-bc78-52bcefbd2aef", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f7e680067d0 con 0x7f7e740cd060 2024-04-25T01:21:22.061 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-04-25T01:21:22.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.080+0000 7f7e78f21640 1 -- 172.21.15.121:0/3721839037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 msgr2=0x7f7e74156fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.080+0000 7f7e78f21640 1 --2- 172.21.15.121:0/3721839037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e74156fd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800c3c0 tx=0x7f7e6800c880 comp rx=0 tx=0).stop 2024-04-25T01:21:22.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.080+0000 7f7e78f21640 1 -- 172.21.15.121:0/3721839037 shutdown_connections 2024-04-25T01:21:22.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.080+0000 7f7e78f21640 1 --2- 172.21.15.121:0/3721839037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e740cf370 0x7f7e74157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.080+0000 7f7e78f21640 1 --2- 172.21.15.121:0/3721839037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e740cd060 0x7f7e74156fd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.080+0000 7f7e78f21640 1 --2- 172.21.15.121:0/3721839037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e740ccc00 0x7f7e740b5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.080+0000 7f7e78f21640 1 -- 172.21.15.121:0/3721839037 >> 172.21.15.121:0/3721839037 conn(0x7f7e740d0bf0 msgr2=0x7f7e7415a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:22.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.080+0000 7f7e78f21640 1 -- 172.21.15.121:0/3721839037 shutdown_connections 2024-04-25T01:21:22.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.080+0000 7f7e78f21640 1 -- 172.21.15.121:0/3721839037 wait complete. 2024-04-25T01:21:22.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 7f461099-df7e-4ae9-bd92-689e529ea919 2 2024-04-25T01:21:22.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.188+0000 7f97ae01c640 1 Processor -- start 2024-04-25T01:21:22.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.188+0000 7f97ae01c640 1 -- start start 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.189+0000 7f97ae01c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f97a80d23d0 0x7f97a80d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.189+0000 7f97ae01c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97a80cea40 con 0x7f97a80d27b0 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.189+0000 7f97ae01c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97a80cece0 con 0x7f97a80b9a90 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.189+0000 7f97ae01c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f97a80cefc0 con 0x7f97a80d23d0 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.189+0000 7f97a7fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f97a80b9a90 0x7f97a80b6c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:42980/0 (socket says 172.21.15.121:42980) 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.189+0000 7f97a7fff640 1 -- 172.21.15.121:0/605627667 learned_addr learned my addr 172.21.15.121:0/605627667 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.189+0000 7f97a77fe640 1 --2- 172.21.15.121:0/605627667 >> v2:172.21.15.77:3300/0 conn(0x7f97a80d23d0 0x7f97a80d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.189+0000 7f97a67fc640 1 -- 172.21.15.121:0/605627667 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 924613030 0 0) 0x7f97a80cece0 con 0x7f97a80b9a90 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.190+0000 7f97a67fc640 1 -- 172.21.15.121:0/605627667 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9790003520 con 0x7f97a80b9a90 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.190+0000 7f97a67fc640 1 -- 172.21.15.121:0/605627667 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1623423175 0 0) 0x7f97a80cea40 con 0x7f97a80d27b0 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.190+0000 7f97a67fc640 1 -- 172.21.15.121:0/605627667 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97a80cece0 con 0x7f97a80d27b0 2024-04-25T01:21:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.190+0000 7f97a77fe640 1 -- 172.21.15.121:0/605627667 >> v1:172.21.15.121:6789/0 conn(0x7f97a80b9a90 legacy=0x7f97a80b6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.190+0000 7f97a77fe640 1 -- 172.21.15.121:0/605627667 >> v1:172.21.15.77:6789/0 conn(0x7f97a80d27b0 legacy=0x7f97a80b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.190+0000 7f97a77fe640 1 -- 172.21.15.121:0/605627667 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a80cf2a0 con 0x7f97a80d23d0 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.190+0000 7f97a77fe640 1 --2- 172.21.15.121:0/605627667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a80d23d0 0x7f97a80d4920 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f97980097a0 tx=0x7f979802a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=679f6f548a101e76 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.191+0000 7f97a67fc640 1 -- 172.21.15.121:0/605627667 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798037070 con 0x7f97a80d23d0 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.191+0000 7f97a67fc640 1 -- 172.21.15.121:0/605627667 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9798032640 con 0x7f97a80d23d0 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.191+0000 7f97a67fc640 1 -- 172.21.15.121:0/605627667 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798032a10 con 0x7f97a80d23d0 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.191+0000 7f97ae01c640 1 -- 172.21.15.121:0/605627667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a80d23d0 msgr2=0x7f97a80d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.191+0000 7f97ae01c640 1 --2- 172.21.15.121:0/605627667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a80d23d0 0x7f97a80d4920 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f97980097a0 tx=0x7f979802a2c0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.191+0000 7f97ae01c640 1 -- 172.21.15.121:0/605627667 shutdown_connections 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.191+0000 7f97ae01c640 1 --2- 172.21.15.121:0/605627667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a80d23d0 0x7f97a80d4920 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.191+0000 7f97ae01c640 1 -- 172.21.15.121:0/605627667 >> 172.21.15.121:0/605627667 conn(0x7f97a80d3e30 msgr2=0x7f97a80d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.191+0000 7f97ae01c640 1 -- 172.21.15.121:0/605627667 shutdown_connections 2024-04-25T01:21:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.191+0000 7f97ae01c640 1 -- 172.21.15.121:0/605627667 wait complete. 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.192+0000 7f97ae01c640 1 Processor -- start 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.192+0000 7f97ae01c640 1 -- start start 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.193+0000 7f97ae01c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97a80b9a90 0x7f97a81554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.193+0000 7f97ae01c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97a80d23d0 0x7f97a80b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.193+0000 7f97ae01c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a80d27b0 0x7f97a80b8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.193+0000 7f97ae01c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f97a80cfd80 con 0x7f97a80d27b0 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.193+0000 7f97ae01c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f97a80cf860 con 0x7f97a80b9a90 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.193+0000 7f97ae01c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f97a80d0ee0 con 0x7f97a80d23d0 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.193+0000 7f97a77fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97a80b9a90 0x7f97a81554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.193+0000 7f97a6ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97a80d23d0 0x7f97a80b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.193+0000 7f97a77fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97a80b9a90 0x7f97a81554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39230/0 (socket says 172.21.15.121:39230) 2024-04-25T01:21:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.193+0000 7f97a77fe640 1 -- 172.21.15.121:0/353312684 learned_addr learned my addr 172.21.15.121:0/353312684 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.193+0000 7f97a7fff640 1 --2- 172.21.15.121:0/353312684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a80d27b0 0x7f97a80b8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f97a77fe640 1 -- 172.21.15.121:0/353312684 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97a80d23d0 msgr2=0x7f97a80b87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f97a77fe640 1 --2- 172.21.15.121:0/353312684 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97a80d23d0 0x7f97a80b87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f97a77fe640 1 -- 172.21.15.121:0/353312684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a80d27b0 msgr2=0x7f97a80b8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f97a77fe640 1 --2- 172.21.15.121:0/353312684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a80d27b0 0x7f97a80b8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f97a77fe640 1 -- 172.21.15.121:0/353312684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a80b7e90 con 0x7f97a80b9a90 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f97a6ffd640 1 --2- 172.21.15.121:0/353312684 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97a80d23d0 0x7f97a80b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f97a77fe640 1 --2- 172.21.15.121:0/353312684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97a80b9a90 0x7f97a81554e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9798035040 tx=0x7f97980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f978ffff640 1 -- 172.21.15.121:0/353312684 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798037040 con 0x7f97a80b9a90 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f978ffff640 1 -- 172.21.15.121:0/353312684 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9798032440 con 0x7f97a80b9a90 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f97ae01c640 1 -- 172.21.15.121:0/353312684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a80b78d0 con 0x7f97a80b9a90 2024-04-25T01:21:22.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f978ffff640 1 -- 172.21.15.121:0/353312684 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798030990 con 0x7f97a80b9a90 2024-04-25T01:21:22.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f97ae01c640 1 -- 172.21.15.121:0/353312684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a8157ef0 con 0x7f97a80b9a90 2024-04-25T01:21:22.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f978ffff640 1 -- 172.21.15.121:0/353312684 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9798044020 con 0x7f97a80b9a90 2024-04-25T01:21:22.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.194+0000 7f978ffff640 1 -- 172.21.15.121:0/353312684 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f979803dc00 con 0x7f97a80b9a90 2024-04-25T01:21:22.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.195+0000 7f97ae01c640 1 -- 172.21.15.121:0/353312684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a8068fb0 con 0x7f97a80b9a90 2024-04-25T01:21:22.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.197+0000 7f978ffff640 1 -- 172.21.15.121:0/353312684 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f97980325f0 con 0x7f97a80b9a90 2024-04-25T01:21:22.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.270+0000 7f97ae01c640 1 -- 172.21.15.121:0/353312684 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7f461099-df7e-4ae9-bd92-689e529ea919", "id": 2} v 0) -- 0x7f97a808df50 con 0x7f97a80b9a90 2024-04-25T01:21:22.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.276+0000 7f978ffff640 1 -- 172.21.15.121:0/353312684 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7f461099-df7e-4ae9-bd92-689e529ea919", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f979803d440 con 0x7f97a80b9a90 2024-04-25T01:21:22.257 INFO:teuthology.orchestra.run.smithi121.stdout:2 2024-04-25T01:21:22.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.277+0000 7f97ae01c640 1 -- 172.21.15.121:0/353312684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97a80b9a90 msgr2=0x7f97a81554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.277+0000 7f97ae01c640 1 --2- 172.21.15.121:0/353312684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97a80b9a90 0x7f97a81554e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9798035040 tx=0x7f97980046c0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.277+0000 7f97ae01c640 1 -- 172.21.15.121:0/353312684 shutdown_connections 2024-04-25T01:21:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.277+0000 7f97ae01c640 1 --2- 172.21.15.121:0/353312684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97a80d27b0 0x7f97a80b8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.277+0000 7f97ae01c640 1 --2- 172.21.15.121:0/353312684 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97a80d23d0 0x7f97a80b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.277+0000 7f97ae01c640 1 --2- 172.21.15.121:0/353312684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97a80b9a90 0x7f97a81554e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.277+0000 7f97ae01c640 1 -- 172.21.15.121:0/353312684 >> 172.21.15.121:0/353312684 conn(0x7f97a80d3e30 msgr2=0x7f97a80cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.277+0000 7f97ae01c640 1 -- 172.21.15.121:0/353312684 shutdown_connections 2024-04-25T01:21:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.277+0000 7f97ae01c640 1 -- 172.21.15.121:0/353312684 wait complete. 2024-04-25T01:21:22.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 1c3b7b96-f9b6-461c-a541-cf96843238d2 3 2024-04-25T01:21:22.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.385+0000 7f2a9b89a640 1 Processor -- start 2024-04-25T01:21:22.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.385+0000 7f2a9b89a640 1 -- start start 2024-04-25T01:21:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a9b89a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a94144880 0x7f2a94071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a9b89a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2a9406e0e0 0x7f2a940b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a9b89a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a9406e6c0 con 0x7f2a94072190 2024-04-25T01:21:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a9b89a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f2a9406e9a0 con 0x7f2a94144880 2024-04-25T01:21:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a9b89a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2a940b3ac0 con 0x7f2a9406e0e0 2024-04-25T01:21:22.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a99e37640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2a9406e0e0 0x7f2a940b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a99e37640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2a9406e0e0 0x7f2a940b45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39238/0 (socket says 172.21.15.121:39238) 2024-04-25T01:21:22.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a99e37640 1 -- 172.21.15.121:0/886670223 learned_addr learned my addr 172.21.15.121:0/886670223 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:22.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a99636640 1 --2- 172.21.15.121:0/886670223 >> v2:172.21.15.77:3300/0 conn(0x7f2a94144880 0x7f2a94071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a99e37640 1 -- 172.21.15.121:0/886670223 >> v2:172.21.15.77:3300/0 conn(0x7f2a94144880 msgr2=0x7f2a94071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:22.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a99e37640 1 --2- 172.21.15.121:0/886670223 >> v2:172.21.15.77:3300/0 conn(0x7f2a94144880 0x7f2a94071c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a99e37640 1 -- 172.21.15.121:0/886670223 >> v1:172.21.15.77:6789/0 conn(0x7f2a94072190 legacy=0x7f2a9406d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.386+0000 7f2a99e37640 1 -- 172.21.15.121:0/886670223 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a940b3da0 con 0x7f2a9406e0e0 2024-04-25T01:21:22.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.387+0000 7f2a99e37640 1 --2- 172.21.15.121:0/886670223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a9406e0e0 0x7f2a940b45a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2a9406fc40 tx=0x7f2a8c02a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5c75b820f76c13a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:22.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.387+0000 7f2a8bfff640 1 -- 172.21.15.121:0/886670223 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c036070 con 0x7f2a9406e0e0 2024-04-25T01:21:22.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.387+0000 7f2a8bfff640 1 -- 172.21.15.121:0/886670223 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a8c031600 con 0x7f2a9406e0e0 2024-04-25T01:21:22.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.387+0000 7f2a8bfff640 1 -- 172.21.15.121:0/886670223 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c0319d0 con 0x7f2a9406e0e0 2024-04-25T01:21:22.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.388+0000 7f2a9b89a640 1 -- 172.21.15.121:0/886670223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a9406e0e0 msgr2=0x7f2a940b45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.388+0000 7f2a9b89a640 1 --2- 172.21.15.121:0/886670223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a9406e0e0 0x7f2a940b45a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2a9406fc40 tx=0x7f2a8c02a6a0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.388+0000 7f2a9b89a640 1 -- 172.21.15.121:0/886670223 shutdown_connections 2024-04-25T01:21:22.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.388+0000 7f2a9b89a640 1 --2- 172.21.15.121:0/886670223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a9406e0e0 0x7f2a940b45a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.388+0000 7f2a9b89a640 1 --2- 172.21.15.121:0/886670223 >> v2:172.21.15.77:3300/0 conn(0x7f2a94144880 0x7f2a94071c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.388+0000 7f2a9b89a640 1 -- 172.21.15.121:0/886670223 >> 172.21.15.121:0/886670223 conn(0x7f2a940d0b70 msgr2=0x7f2a940d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:22.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.388+0000 7f2a9b89a640 1 -- 172.21.15.121:0/886670223 shutdown_connections 2024-04-25T01:21:22.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.388+0000 7f2a9b89a640 1 -- 172.21.15.121:0/886670223 wait complete. 2024-04-25T01:21:22.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.389+0000 7f2a9b89a640 1 Processor -- start 2024-04-25T01:21:22.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.389+0000 7f2a9b89a640 1 -- start start 2024-04-25T01:21:22.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a9b89a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a9406e0e0 0x7f2a94156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a9b89a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a94072190 0x7f2a940afed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a9b89a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a94144880 0x7f2a94158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a9b89a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2a940b4e70 con 0x7f2a9406e0e0 2024-04-25T01:21:22.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a9b89a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2a940b4cf0 con 0x7f2a94144880 2024-04-25T01:21:22.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a9b89a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a940b4ff0 con 0x7f2a94072190 2024-04-25T01:21:22.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a98e35640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a94072190 0x7f2a940afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a99e37640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a94144880 0x7f2a94158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a99636640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a9406e0e0 0x7f2a94156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a99e37640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a94144880 0x7f2a94158700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39254/0 (socket says 172.21.15.121:39254) 2024-04-25T01:21:22.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a99e37640 1 -- 172.21.15.121:0/4288578601 learned_addr learned my addr 172.21.15.121:0/4288578601 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:22.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a98e35640 1 --2- 172.21.15.121:0/4288578601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a94072190 0x7f2a940afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37038/0 (socket says 172.21.15.121:37038) 2024-04-25T01:21:22.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a99636640 1 -- 172.21.15.121:0/4288578601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a94072190 msgr2=0x7f2a940afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a99636640 1 --2- 172.21.15.121:0/4288578601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a94072190 0x7f2a940afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a99636640 1 -- 172.21.15.121:0/4288578601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a94144880 msgr2=0x7f2a94158700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a99636640 1 --2- 172.21.15.121:0/4288578601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a94144880 0x7f2a94158700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.390+0000 7f2a99636640 1 -- 172.21.15.121:0/4288578601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a94158c40 con 0x7f2a9406e0e0 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.391+0000 7f2a99636640 1 --2- 172.21.15.121:0/4288578601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a9406e0e0 0x7f2a94156bd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000e870 tx=0x7f2a9000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.391+0000 7f2a89ffb640 1 -- 172.21.15.121:0/4288578601 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a90017070 con 0x7f2a9406e0e0 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.391+0000 7f2a89ffb640 1 -- 172.21.15.121:0/4288578601 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a9000ee60 con 0x7f2a9406e0e0 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.391+0000 7f2a89ffb640 1 -- 172.21.15.121:0/4288578601 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a9000cb60 con 0x7f2a9406e0e0 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.391+0000 7f2a9b89a640 1 -- 172.21.15.121:0/4288578601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a941571b0 con 0x7f2a9406e0e0 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.391+0000 7f2a9b89a640 1 -- 172.21.15.121:0/4288578601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a94157770 con 0x7f2a9406e0e0 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.391+0000 7f2a89ffb640 1 -- 172.21.15.121:0/4288578601 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2a9001a070 con 0x7f2a9406e0e0 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.391+0000 7f2a89ffb640 1 -- 172.21.15.121:0/4288578601 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f2a9001c8e0 con 0x7f2a9406e0e0 2024-04-25T01:21:22.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.392+0000 7f2a9b89a640 1 -- 172.21.15.121:0/4288578601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a940b36d0 con 0x7f2a9406e0e0 2024-04-25T01:21:22.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.396+0000 7f2a89ffb640 1 -- 172.21.15.121:0/4288578601 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2a940b36d0 con 0x7f2a9406e0e0 2024-04-25T01:21:22.388 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:22.408+0000 7f4756320040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:21:22.426 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:22.445+0000 7fad55410040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:21:22.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.468+0000 7f2a9b89a640 1 -- 172.21.15.121:0/4288578601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1c3b7b96-f9b6-461c-a541-cf96843238d2", "id": 3} v 0) -- 0x7f2a94068180 con 0x7f2a9406e0e0 2024-04-25T01:21:22.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.472+0000 7f2a89ffb640 1 -- 172.21.15.121:0/4288578601 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1c3b7b96-f9b6-461c-a541-cf96843238d2", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f2a9001cbd0 con 0x7f2a9406e0e0 2024-04-25T01:21:22.454 INFO:teuthology.orchestra.run.smithi121.stdout:3 2024-04-25T01:21:22.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.473+0000 7f2a9b89a640 1 -- 172.21.15.121:0/4288578601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a9406e0e0 msgr2=0x7f2a94156bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.473+0000 7f2a9b89a640 1 --2- 172.21.15.121:0/4288578601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a9406e0e0 0x7f2a94156bd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000e870 tx=0x7f2a9000ed30 comp rx=0 tx=0).stop 2024-04-25T01:21:22.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.473+0000 7f2a9b89a640 1 -- 172.21.15.121:0/4288578601 shutdown_connections 2024-04-25T01:21:22.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.473+0000 7f2a9b89a640 1 --2- 172.21.15.121:0/4288578601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a94144880 0x7f2a94158700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.473+0000 7f2a9b89a640 1 --2- 172.21.15.121:0/4288578601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a94072190 0x7f2a940afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.473+0000 7f2a9b89a640 1 --2- 172.21.15.121:0/4288578601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a9406e0e0 0x7f2a94156bd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.473+0000 7f2a9b89a640 1 -- 172.21.15.121:0/4288578601 >> 172.21.15.121:0/4288578601 conn(0x7f2a940d0b70 msgr2=0x7f2a9415f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:22.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.473+0000 7f2a9b89a640 1 -- 172.21.15.121:0/4288578601 shutdown_connections 2024-04-25T01:21:22.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.473+0000 7f2a9b89a640 1 -- 172.21.15.121:0/4288578601 wait complete. 2024-04-25T01:21:22.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 17a4c982-a80b-4b7d-8964-b8b476bca624 4 2024-04-25T01:21:22.475 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:22.494+0000 7f4756320040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:21:22.518 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:22.537+0000 7f4756320040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:21:22.520 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:22.538+0000 7fad55410040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:21:22.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.580+0000 7fc2509f9640 1 Processor -- start 2024-04-25T01:21:22.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.580+0000 7fc2509f9640 1 -- start start 2024-04-25T01:21:22.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.581+0000 7fc2509f9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc24c1478a0 0x7fc24c072b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.581+0000 7fc2509f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06ec40 0x7fc24c06b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.581+0000 7fc2509f9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc24c06bbc0 con 0x7fc24c06b120 2024-04-25T01:21:22.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.581+0000 7fc2509f9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc24c071da0 con 0x7fc24c1478a0 2024-04-25T01:21:22.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.581+0000 7fc2509f9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc24c072080 con 0x7fc24c06ec40 2024-04-25T01:21:22.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.581+0000 7fc24a59c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc24c1478a0 0x7fc24c072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.581+0000 7fc24ad9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06ec40 0x7fc24c06b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.581+0000 7fc243fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc24c06b120 0x7fc24c06e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:42992/0 (socket says 172.21.15.121:42992) 2024-04-25T01:21:22.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.581+0000 7fc243fff640 1 -- 172.21.15.121:0/1984940866 learned_addr learned my addr 172.21.15.121:0/1984940866 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-04-25T01:21:22.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.582+0000 7fc24a59c640 1 -- 172.21.15.121:0/1984940866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06ec40 msgr2=0x7fc24c06b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:22.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.582+0000 7fc24a59c640 1 --2- 172.21.15.121:0/1984940866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06ec40 0x7fc24c06b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.582+0000 7fc24a59c640 1 -- 172.21.15.121:0/1984940866 >> v1:172.21.15.121:6789/0 conn(0x7fc24c06b120 legacy=0x7fc24c06e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.582+0000 7fc24a59c640 1 -- 172.21.15.121:0/1984940866 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c072360 con 0x7fc24c1478a0 2024-04-25T01:21:22.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.582+0000 7fc24ad9d640 1 --2- 172.21.15.121:0/1984940866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06ec40 0x7fc24c06b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:22.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.582+0000 7fc24a59c640 1 --2- 172.21.15.121:0/1984940866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc24c1478a0 0x7fc24c072b60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0097a0 tx=0x7fc23c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dd1bd9032a38693 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:22.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.582+0000 7fc249d9b640 1 -- 172.21.15.121:0/1984940866 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c037070 con 0x7fc24c1478a0 2024-04-25T01:21:22.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.582+0000 7fc249d9b640 1 -- 172.21.15.121:0/1984940866 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc23c032440 con 0x7fc24c1478a0 2024-04-25T01:21:22.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.582+0000 7fc249d9b640 1 -- 172.21.15.121:0/1984940866 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c032810 con 0x7fc24c1478a0 2024-04-25T01:21:22.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.583+0000 7fc2509f9640 1 -- 172.21.15.121:0/1984940866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc24c1478a0 msgr2=0x7fc24c072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.583+0000 7fc2509f9640 1 --2- 172.21.15.121:0/1984940866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc24c1478a0 0x7fc24c072b60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0097a0 tx=0x7fc23c02a180 comp rx=0 tx=0).stop 2024-04-25T01:21:22.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.583+0000 7fc2509f9640 1 -- 172.21.15.121:0/1984940866 shutdown_connections 2024-04-25T01:21:22.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.583+0000 7fc2509f9640 1 --2- 172.21.15.121:0/1984940866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06ec40 0x7fc24c06b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.583+0000 7fc2509f9640 1 --2- 172.21.15.121:0/1984940866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc24c1478a0 0x7fc24c072b60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.583+0000 7fc2509f9640 1 -- 172.21.15.121:0/1984940866 >> 172.21.15.121:0/1984940866 conn(0x7fc24c0d0bf0 msgr2=0x7fc24c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:22.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.583+0000 7fc2509f9640 1 -- 172.21.15.121:0/1984940866 shutdown_connections 2024-04-25T01:21:22.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.583+0000 7fc2509f9640 1 -- 172.21.15.121:0/1984940866 wait complete. 2024-04-25T01:21:22.568 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:22.583+0000 7fad55410040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:21:22.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.584+0000 7fc2509f9640 1 Processor -- start 2024-04-25T01:21:22.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.584+0000 7fc2509f9640 1 -- start start 2024-04-25T01:21:22.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc2509f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06b120 0x7fc24c13f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc2509f9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc24c06ec40 0x7fc24c141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc2509f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c1478a0 0x7fc24c142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc2509f9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c162900 con 0x7fc24c1478a0 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc2509f9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c0cef60 con 0x7fc24c06ec40 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc2509f9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc24c1f7740 con 0x7fc24c06b120 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc24a59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06b120 0x7fc24c13f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc243fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc24c06ec40 0x7fc24c141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc24a59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06b120 0x7fc24c13f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37050/0 (socket says 172.21.15.121:37050) 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc243fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc24c06ec40 0x7fc24c141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39272/0 (socket says 172.21.15.121:39272) 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc24a59c640 1 -- 172.21.15.121:0/698366645 learned_addr learned my addr 172.21.15.121:0/698366645 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.585+0000 7fc24ad9d640 1 --2- 172.21.15.121:0/698366645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c1478a0 0x7fc24c142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc24a59c640 1 -- 172.21.15.121:0/698366645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc24c06ec40 msgr2=0x7fc24c141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc24a59c640 1 --2- 172.21.15.121:0/698366645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc24c06ec40 0x7fc24c141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc24a59c640 1 -- 172.21.15.121:0/698366645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c1478a0 msgr2=0x7fc24c142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc24a59c640 1 --2- 172.21.15.121:0/698366645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c1478a0 0x7fc24c142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc24a59c640 1 -- 172.21.15.121:0/698366645 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c1f7a20 con 0x7fc24c06b120 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc243fff640 1 --2- 172.21.15.121:0/698366645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc24c06ec40 0x7fc24c141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc24a59c640 1 --2- 172.21.15.121:0/698366645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06b120 0x7fc24c13f800 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc23c035040 tx=0x7fc23c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc242ffd640 1 -- 172.21.15.121:0/698366645 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c037040 con 0x7fc24c06b120 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc242ffd640 1 -- 172.21.15.121:0/698366645 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc23c030440 con 0x7fc24c06b120 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc242ffd640 1 -- 172.21.15.121:0/698366645 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c030850 con 0x7fc24c06b120 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc2509f9640 1 -- 172.21.15.121:0/698366645 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24c1f7d00 con 0x7fc24c06b120 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc2509f9640 1 -- 172.21.15.121:0/698366645 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24c1e7920 con 0x7fc24c06b120 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc242ffd640 1 -- 172.21.15.121:0/698366645 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc23c044020 con 0x7fc24c06b120 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.586+0000 7fc242ffd640 1 -- 172.21.15.121:0/698366645 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fc23c03d420 con 0x7fc24c06b120 2024-04-25T01:21:22.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.587+0000 7fc2509f9640 1 -- 172.21.15.121:0/698366645 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc24c068180 con 0x7fc24c06b120 2024-04-25T01:21:22.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.589+0000 7fc242ffd640 1 -- 172.21.15.121:0/698366645 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc23c03b430 con 0x7fc24c06b120 2024-04-25T01:21:22.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.682+0000 7fc2509f9640 1 -- 172.21.15.121:0/698366645 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "17a4c982-a80b-4b7d-8964-b8b476bca624", "id": 4} v 0) -- 0x7fc24c06be00 con 0x7fc24c06b120 2024-04-25T01:21:22.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.687+0000 7fc242ffd640 1 -- 172.21.15.121:0/698366645 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "17a4c982-a80b-4b7d-8964-b8b476bca624", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fc23c03b610 con 0x7fc24c06b120 2024-04-25T01:21:22.669 INFO:teuthology.orchestra.run.smithi121.stdout:4 2024-04-25T01:21:22.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.688+0000 7fc2509f9640 1 -- 172.21.15.121:0/698366645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06b120 msgr2=0x7fc24c13f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.688+0000 7fc2509f9640 1 --2- 172.21.15.121:0/698366645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06b120 0x7fc24c13f800 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc23c035040 tx=0x7fc23c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.688+0000 7fc2509f9640 1 -- 172.21.15.121:0/698366645 shutdown_connections 2024-04-25T01:21:22.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.688+0000 7fc2509f9640 1 --2- 172.21.15.121:0/698366645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c1478a0 0x7fc24c142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.688+0000 7fc2509f9640 1 --2- 172.21.15.121:0/698366645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc24c06ec40 0x7fc24c141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.688+0000 7fc2509f9640 1 --2- 172.21.15.121:0/698366645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc24c06b120 0x7fc24c13f800 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.688+0000 7fc2509f9640 1 -- 172.21.15.121:0/698366645 >> 172.21.15.121:0/698366645 conn(0x7fc24c0d0bf0 msgr2=0x7fc24c0cbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:22.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.688+0000 7fc2509f9640 1 -- 172.21.15.121:0/698366645 shutdown_connections 2024-04-25T01:21:22.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.688+0000 7fc2509f9640 1 -- 172.21.15.121:0/698366645 wait complete. 2024-04-25T01:21:22.680 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 13fc3195-286c-4f2d-9d9d-e48d1adb9294 5 2024-04-25T01:21:22.681 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:22.700+0000 7f4756320040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:21:22.729 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:22.748+0000 7f4756320040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:21:22.757 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:22.775+0000 7fad55410040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:21:22.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.796+0000 7fb7b5ef5640 1 Processor -- start 2024-04-25T01:21:22.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.796+0000 7fb7b5ef5640 1 -- start start 2024-04-25T01:21:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7b5ef5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b006b830 0x7fb7b00d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7b5ef5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb7b0147440 0x7fb7b0071d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7b5ef5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b00cee30 con 0x7fb7b0144970 2024-04-25T01:21:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7b5ef5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb7b00cadf0 con 0x7fb7b0147440 2024-04-25T01:21:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7b5ef5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b00c9f40 con 0x7fb7b006b830 2024-04-25T01:21:22.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7a7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b006b830 0x7fb7b00d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7affff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb7b0147440 0x7fb7b0071d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7affff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb7b0147440 0x7fb7b0071d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39284/0 (socket says 172.21.15.121:39284) 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7affff640 1 -- 172.21.15.121:0/2306068858 learned_addr learned my addr 172.21.15.121:0/2306068858 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7a7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b006b830 0x7fb7b00d1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37058/0 (socket says 172.21.15.121:37058) 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7a7fff640 1 -- 172.21.15.121:0/2306068858 >> v2:172.21.15.121:3300/0 conn(0x7fb7b0147440 msgr2=0x7fb7b0071d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7a7fff640 1 --2- 172.21.15.121:0/2306068858 >> v2:172.21.15.121:3300/0 conn(0x7fb7b0147440 0x7fb7b0071d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7a7fff640 1 -- 172.21.15.121:0/2306068858 >> v1:172.21.15.77:6789/0 conn(0x7fb7b0144970 legacy=0x7fb7b006b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7a7fff640 1 -- 172.21.15.121:0/2306068858 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b00ca1e0 con 0x7fb7b006b830 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.797+0000 7fb7affff640 1 --2- 172.21.15.121:0/2306068858 >> v2:172.21.15.121:3300/0 conn(0x7fb7b0147440 0x7fb7b0071d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.798+0000 7fb7a7fff640 1 --2- 172.21.15.121:0/2306068858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b006b830 0x7fb7b00d1710 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb7980097a0 tx=0x7fb79802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e004c65132fcf55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.798+0000 7fb7aeffd640 1 -- 172.21.15.121:0/2306068858 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb798037070 con 0x7fb7b006b830 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.798+0000 7fb7aeffd640 1 -- 172.21.15.121:0/2306068858 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb798032440 con 0x7fb7b006b830 2024-04-25T01:21:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.798+0000 7fb7aeffd640 1 -- 172.21.15.121:0/2306068858 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb798032810 con 0x7fb7b006b830 2024-04-25T01:21:22.782 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:22.800+0000 7f4756320040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:21:22.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.799+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2306068858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b006b830 msgr2=0x7fb7b00d1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.799+0000 7fb7b5ef5640 1 --2- 172.21.15.121:0/2306068858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b006b830 0x7fb7b00d1710 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb7980097a0 tx=0x7fb79802a180 comp rx=0 tx=0).stop 2024-04-25T01:21:22.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.799+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2306068858 shutdown_connections 2024-04-25T01:21:22.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.799+0000 7fb7b5ef5640 1 --2- 172.21.15.121:0/2306068858 >> v2:172.21.15.121:3300/0 conn(0x7fb7b0147440 0x7fb7b0071d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.799+0000 7fb7b5ef5640 1 --2- 172.21.15.121:0/2306068858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b006b830 0x7fb7b00d1710 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.799+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2306068858 >> 172.21.15.121:0/2306068858 conn(0x7fb7b00d0bf0 msgr2=0x7fb7b00d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:22.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.799+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2306068858 shutdown_connections 2024-04-25T01:21:22.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.799+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2306068858 wait complete. 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.800+0000 7fb7b5ef5640 1 Processor -- start 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.800+0000 7fb7b5ef5640 1 -- start start 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7b5ef5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7b006b830 0x7fb7b00b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7b5ef5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b0144970 0x7fb7b00cc980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7b5ef5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b0147440 0x7fb7b00b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7b5ef5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b00caa80 con 0x7fb7b0144970 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7b5ef5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b00ca7a0 con 0x7fb7b006b830 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7b5ef5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b00722d0 con 0x7fb7b0147440 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7affff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b0147440 0x7fb7b00b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7af7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7b006b830 0x7fb7b00b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7af7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7b006b830 0x7fb7b00b5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39294/0 (socket says 172.21.15.121:39294) 2024-04-25T01:21:22.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7affff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b0147440 0x7fb7b00b3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37066/0 (socket says 172.21.15.121:37066) 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7af7fe640 1 -- 172.21.15.121:0/2402910711 learned_addr learned my addr 172.21.15.121:0/2402910711 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7a7fff640 1 --2- 172.21.15.121:0/2402910711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b0144970 0x7fb7b00cc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7affff640 1 -- 172.21.15.121:0/2402910711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7b006b830 msgr2=0x7fb7b00b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7affff640 1 --2- 172.21.15.121:0/2402910711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7b006b830 0x7fb7b00b5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7affff640 1 -- 172.21.15.121:0/2402910711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b0144970 msgr2=0x7fb7b00cc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7affff640 1 --2- 172.21.15.121:0/2402910711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b0144970 0x7fb7b00cc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7affff640 1 -- 172.21.15.121:0/2402910711 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b01404b0 con 0x7fb7b0147440 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7af7fe640 1 --2- 172.21.15.121:0/2402910711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7b006b830 0x7fb7b00b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7a7fff640 1 --2- 172.21.15.121:0/2402910711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b0144970 0x7fb7b00cc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.801+0000 7fb7affff640 1 --2- 172.21.15.121:0/2402910711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b0147440 0x7fb7b00b3980 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800ef90 tx=0x7fb7a800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.802+0000 7fb7acff9640 1 -- 172.21.15.121:0/2402910711 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a800cd20 con 0x7fb7b0147440 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.802+0000 7fb7acff9640 1 -- 172.21.15.121:0/2402910711 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7a8019070 con 0x7fb7b0147440 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.802+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2402910711 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b00b44f0 con 0x7fb7b0147440 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.802+0000 7fb7acff9640 1 -- 172.21.15.121:0/2402910711 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a80146c0 con 0x7fb7b0147440 2024-04-25T01:21:22.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.802+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2402910711 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b00b4d90 con 0x7fb7b0147440 2024-04-25T01:21:22.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.802+0000 7fb7acff9640 1 -- 172.21.15.121:0/2402910711 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb7a8005d10 con 0x7fb7b0147440 2024-04-25T01:21:22.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.802+0000 7fb7acff9640 1 -- 172.21.15.121:0/2402910711 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fb7a8009a30 con 0x7fb7b0147440 2024-04-25T01:21:22.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.803+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2402910711 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7b0068180 con 0x7fb7b0147440 2024-04-25T01:21:22.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.806+0000 7fb7acff9640 1 -- 172.21.15.121:0/2402910711 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb7a8010040 con 0x7fb7b0147440 2024-04-25T01:21:22.827 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:22.846+0000 7fad55410040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:21:22.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.879+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2402910711 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "13fc3195-286c-4f2d-9d9d-e48d1adb9294", "id": 5} v 0) -- 0x7fb7b006da80 con 0x7fb7b0147440 2024-04-25T01:21:22.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.884+0000 7fb7acff9640 1 -- 172.21.15.121:0/2402910711 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "13fc3195-286c-4f2d-9d9d-e48d1adb9294", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fb7a801d410 con 0x7fb7b0147440 2024-04-25T01:21:22.866 INFO:teuthology.orchestra.run.smithi121.stdout:5 2024-04-25T01:21:22.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.885+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2402910711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b0147440 msgr2=0x7fb7b00b3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.885+0000 7fb7b5ef5640 1 --2- 172.21.15.121:0/2402910711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b0147440 0x7fb7b00b3980 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800ef90 tx=0x7fb7a800c490 comp rx=0 tx=0).stop 2024-04-25T01:21:22.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.885+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2402910711 shutdown_connections 2024-04-25T01:21:22.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.885+0000 7fb7b5ef5640 1 --2- 172.21.15.121:0/2402910711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b0144970 0x7fb7b00cc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.885+0000 7fb7b5ef5640 1 --2- 172.21.15.121:0/2402910711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b0147440 0x7fb7b00b3980 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.885+0000 7fb7b5ef5640 1 --2- 172.21.15.121:0/2402910711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7b006b830 0x7fb7b00b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.885+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2402910711 >> 172.21.15.121:0/2402910711 conn(0x7fb7b00d0bf0 msgr2=0x7fb7b00cb360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:22.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.885+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2402910711 shutdown_connections 2024-04-25T01:21:22.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.885+0000 7fb7b5ef5640 1 -- 172.21.15.121:0/2402910711 wait complete. 2024-04-25T01:21:22.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 7c2ee0f6-6136-4fde-89a2-5be0ba7879b3 6 2024-04-25T01:21:22.884 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:22.902+0000 7fad55410040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:21:22.892 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:22.911+0000 7f4756320040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:21:22.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.992+0000 7fb4e3876640 1 Processor -- start 2024-04-25T01:21:22.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.993+0000 7fb4e3876640 1 -- start start 2024-04-25T01:21:22.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.993+0000 7fb4e3876640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb4dc0b42c0 0x7fb4dc06cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.993+0000 7fb4e3876640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4dc0d0a90 con 0x7fb4dc0b2920 2024-04-25T01:21:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.993+0000 7fb4e3876640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4dc0d0d70 con 0x7fb4dc161df0 2024-04-25T01:21:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.993+0000 7fb4e3876640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb4dc162d00 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.994+0000 7fb4e1e13640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb4dc0b42c0 0x7fb4dc06cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.994+0000 7fb4e1e13640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb4dc0b42c0 0x7fb4dc06cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39308/0 (socket says 172.21.15.121:39308) 2024-04-25T01:21:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.994+0000 7fb4e1612640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb4dc161df0 0x7fb4dc1621d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:43004/0 (socket says 172.21.15.121:43004) 2024-04-25T01:21:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.994+0000 7fb4e1e13640 1 -- 172.21.15.121:0/3702812622 learned_addr learned my addr 172.21.15.121:0/3702812622 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.994+0000 7fb4e1e13640 1 -- 172.21.15.121:0/3702812622 >> v1:172.21.15.121:6789/0 conn(0x7fb4dc161df0 legacy=0x7fb4dc1621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.994+0000 7fb4e1e13640 1 -- 172.21.15.121:0/3702812622 >> v1:172.21.15.77:6789/0 conn(0x7fb4dc0b2920 legacy=0x7fb4dc06c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.994+0000 7fb4d3fff640 1 -- 172.21.15.121:0/3702812622 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2794459780 0 0) 0x7fb4dc0d0d70 con 0x7fb4dc161df0 2024-04-25T01:21:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.994+0000 7fb4e1e13640 1 -- 172.21.15.121:0/3702812622 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4dc0d1050 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.995+0000 7fb4e1e13640 1 --2- 172.21.15.121:0/3702812622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4dc0b42c0 0x7fb4dc06cd20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4009e50 tx=0x7fb4d402a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9395c38e4c89ea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:22.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.995+0000 7fb4d3fff640 1 -- 172.21.15.121:0/3702812622 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4036070 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.995+0000 7fb4d3fff640 1 -- 172.21.15.121:0/3702812622 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4d4031600 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.995+0000 7fb4d3fff640 1 -- 172.21.15.121:0/3702812622 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d40319d0 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.996+0000 7fb4e3876640 1 -- 172.21.15.121:0/3702812622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4dc0b42c0 msgr2=0x7fb4dc06cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.996+0000 7fb4e3876640 1 --2- 172.21.15.121:0/3702812622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4dc0b42c0 0x7fb4dc06cd20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4009e50 tx=0x7fb4d402a6a0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.996+0000 7fb4e3876640 1 -- 172.21.15.121:0/3702812622 shutdown_connections 2024-04-25T01:21:22.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.996+0000 7fb4e3876640 1 --2- 172.21.15.121:0/3702812622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4dc0b42c0 0x7fb4dc06cd20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.996+0000 7fb4e3876640 1 -- 172.21.15.121:0/3702812622 >> 172.21.15.121:0/3702812622 conn(0x7fb4dc0b37a0 msgr2=0x7fb4dc0b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:22.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.996+0000 7fb4e3876640 1 -- 172.21.15.121:0/3702812622 shutdown_connections 2024-04-25T01:21:22.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.996+0000 7fb4e3876640 1 -- 172.21.15.121:0/3702812622 wait complete. 2024-04-25T01:21:22.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.997+0000 7fb4e3876640 1 Processor -- start 2024-04-25T01:21:22.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.998+0000 7fb4e3876640 1 -- start start 2024-04-25T01:21:22.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e3876640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4dc0b2920 0x7fb4dc1e8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e3876640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4dc0b42c0 0x7fb4dc0cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e3876640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4dc161df0 0x7fb4dc0cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:22.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e3876640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb4dc162a60 con 0x7fb4dc0b2920 2024-04-25T01:21:22.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e3876640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb4dc1628e0 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e3876640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb4dc0aea80 con 0x7fb4dc161df0 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e0e11640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4dc0b42c0 0x7fb4dc0cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e1e13640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4dc161df0 0x7fb4dc0cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e0e11640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4dc0b42c0 0x7fb4dc0cafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39310/0 (socket says 172.21.15.121:39310) 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e0e11640 1 -- 172.21.15.121:0/4262558607 learned_addr learned my addr 172.21.15.121:0/4262558607 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e1612640 1 --2- 172.21.15.121:0/4262558607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4dc0b2920 0x7fb4dc1e8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e0e11640 1 -- 172.21.15.121:0/4262558607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4dc161df0 msgr2=0x7fb4dc0cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e0e11640 1 --2- 172.21.15.121:0/4262558607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4dc161df0 0x7fb4dc0cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e0e11640 1 -- 172.21.15.121:0/4262558607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4dc0b2920 msgr2=0x7fb4dc1e8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e0e11640 1 --2- 172.21.15.121:0/4262558607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4dc0b2920 0x7fb4dc1e8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e0e11640 1 -- 172.21.15.121:0/4262558607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4dc0cbc60 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e1e13640 1 --2- 172.21.15.121:0/4262558607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4dc161df0 0x7fb4dc0cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4e0e11640 1 --2- 172.21.15.121:0/4262558607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4dc0b42c0 0x7fb4dc0cafe0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00c480 tx=0x7fb4cc00c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:22.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:22.999+0000 7fb4d1ffb640 1 -- 172.21.15.121:0/4262558607 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc0052d0 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.000+0000 7fb4d1ffb640 1 -- 172.21.15.121:0/4262558607 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4cc005470 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.000+0000 7fb4e3876640 1 -- 172.21.15.121:0/4262558607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4dc0cb980 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.000+0000 7fb4d1ffb640 1 -- 172.21.15.121:0/4262558607 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc012870 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.000+0000 7fb4d1ffb640 1 -- 172.21.15.121:0/4262558607 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb4cc012a10 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.000+0000 7fb4e3876640 1 -- 172.21.15.121:0/4262558607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4dc0cc6b0 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.000+0000 7fb4d1ffb640 1 -- 172.21.15.121:0/4262558607 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fb4cc003be0 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.001+0000 7fb4e3876640 1 -- 172.21.15.121:0/4262558607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4dc068180 con 0x7fb4dc0b42c0 2024-04-25T01:21:22.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.003+0000 7fb4d1ffb640 1 -- 172.21.15.121:0/4262558607 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb4cc0035f0 con 0x7fb4dc0b42c0 2024-04-25T01:21:23.005 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:23.023+0000 7fad55410040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:21:23.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.074+0000 7fb4e3876640 1 -- 172.21.15.121:0/4262558607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7c2ee0f6-6136-4fde-89a2-5be0ba7879b3", "id": 6} v 0) -- 0x7fb4dc0b7bf0 con 0x7fb4dc0b42c0 2024-04-25T01:21:23.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.080+0000 7fb4d1ffb640 1 -- 172.21.15.121:0/4262558607 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7c2ee0f6-6136-4fde-89a2-5be0ba7879b3", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fb4cc0037d0 con 0x7fb4dc0b42c0 2024-04-25T01:21:23.062 INFO:teuthology.orchestra.run.smithi121.stdout:6 2024-04-25T01:21:23.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.081+0000 7fb4e3876640 1 -- 172.21.15.121:0/4262558607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4dc0b42c0 msgr2=0x7fb4dc0cafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:23.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.081+0000 7fb4e3876640 1 --2- 172.21.15.121:0/4262558607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4dc0b42c0 0x7fb4dc0cafe0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00c480 tx=0x7fb4cc00c940 comp rx=0 tx=0).stop 2024-04-25T01:21:23.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.081+0000 7fb4e3876640 1 -- 172.21.15.121:0/4262558607 shutdown_connections 2024-04-25T01:21:23.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.081+0000 7fb4e3876640 1 --2- 172.21.15.121:0/4262558607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4dc161df0 0x7fb4dc0cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.081+0000 7fb4e3876640 1 --2- 172.21.15.121:0/4262558607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4dc0b42c0 0x7fb4dc0cafe0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.081+0000 7fb4e3876640 1 --2- 172.21.15.121:0/4262558607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4dc0b2920 0x7fb4dc1e8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.081+0000 7fb4e3876640 1 -- 172.21.15.121:0/4262558607 >> 172.21.15.121:0/4262558607 conn(0x7fb4dc0b37a0 msgr2=0x7fb4dc0c5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:23.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.081+0000 7fb4e3876640 1 -- 172.21.15.121:0/4262558607 shutdown_connections 2024-04-25T01:21:23.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.081+0000 7fb4e3876640 1 -- 172.21.15.121:0/4262558607 wait complete. 2024-04-25T01:21:23.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 79293f60-29fe-4279-90a3-5d95cd553b00 7 2024-04-25T01:21:23.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.228+0000 7fdea7015640 1 Processor -- start 2024-04-25T01:21:23.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.228+0000 7fdea7015640 1 -- start start 2024-04-25T01:21:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea7015640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdea00b4ac0 0x7fdea0160300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea7015640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 0x7fdea00cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea7015640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea0160840 con 0x7fdea015f6a0 2024-04-25T01:21:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea7015640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fdea0160b00 con 0x7fdea00b4ac0 2024-04-25T01:21:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea7015640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdea00cf840 con 0x7fdea00b2be0 2024-04-25T01:21:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea55b2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 0x7fdea00cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea4db1640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fdea015f6a0 0x7fdea015fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:43006/0 (socket says 172.21.15.121:43006) 2024-04-25T01:21:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea4db1640 1 -- 172.21.15.121:0/578802752 learned_addr learned my addr 172.21.15.121:0/578802752 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-04-25T01:21:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fde9ffff640 1 --2- 172.21.15.121:0/578802752 >> v2:172.21.15.77:3300/0 conn(0x7fdea00b4ac0 0x7fdea0160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fde9f7fe640 1 -- 172.21.15.121:0/578802752 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3638582357 0 0) 0x7fdea0160840 con 0x7fdea015f6a0 2024-04-25T01:21:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fde9f7fe640 1 -- 172.21.15.121:0/578802752 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde90003520 con 0x7fdea015f6a0 2024-04-25T01:21:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea55b2640 1 -- 172.21.15.121:0/578802752 >> v2:172.21.15.77:3300/0 conn(0x7fdea00b4ac0 msgr2=0x7fdea0160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea55b2640 1 --2- 172.21.15.121:0/578802752 >> v2:172.21.15.77:3300/0 conn(0x7fdea00b4ac0 0x7fdea0160300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea55b2640 1 -- 172.21.15.121:0/578802752 >> v1:172.21.15.121:6789/0 conn(0x7fdea015f6a0 legacy=0x7fdea015fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.229+0000 7fdea55b2640 1 -- 172.21.15.121:0/578802752 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea00cd380 con 0x7fdea00b2be0 2024-04-25T01:21:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.230+0000 7fdea55b2640 1 --2- 172.21.15.121:0/578802752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 0x7fdea00cfcd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fde98009d90 tx=0x7fde9802a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6b815e4246d5c3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.230+0000 7fde9f7fe640 1 -- 172.21.15.121:0/578802752 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde98007740 con 0x7fdea00b2be0 2024-04-25T01:21:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.230+0000 7fde9f7fe640 1 -- 172.21.15.121:0/578802752 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde980078e0 con 0x7fdea00b2be0 2024-04-25T01:21:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.230+0000 7fde9f7fe640 1 -- 172.21.15.121:0/578802752 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde98007cb0 con 0x7fdea00b2be0 2024-04-25T01:21:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.231+0000 7fdea7015640 1 -- 172.21.15.121:0/578802752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 msgr2=0x7fdea00cfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.231+0000 7fdea7015640 1 --2- 172.21.15.121:0/578802752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 0x7fdea00cfcd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fde98009d90 tx=0x7fde9802a530 comp rx=0 tx=0).stop 2024-04-25T01:21:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.231+0000 7fdea7015640 1 -- 172.21.15.121:0/578802752 shutdown_connections 2024-04-25T01:21:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.231+0000 7fdea7015640 1 --2- 172.21.15.121:0/578802752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 0x7fdea00cfcd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.231+0000 7fdea7015640 1 --2- 172.21.15.121:0/578802752 >> v2:172.21.15.77:3300/0 conn(0x7fdea00b4ac0 0x7fdea0160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.231+0000 7fdea7015640 1 -- 172.21.15.121:0/578802752 >> 172.21.15.121:0/578802752 conn(0x7fdea00b37a0 msgr2=0x7fdea00b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.231+0000 7fdea7015640 1 -- 172.21.15.121:0/578802752 shutdown_connections 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.231+0000 7fdea7015640 1 -- 172.21.15.121:0/578802752 wait complete. 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.232+0000 7fdea7015640 1 Processor -- start 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.232+0000 7fdea7015640 1 -- start start 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fdea7015640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 0x7fdea014fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fdea7015640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdea00b4ac0 0x7fdea006cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fdea7015640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdea015f6a0 0x7fdea00c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fdea7015640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdea00b42c0 con 0x7fdea015f6a0 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fdea7015640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdea00cd940 con 0x7fdea00b4ac0 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fdea7015640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdea00b4440 con 0x7fdea00b2be0 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fde9ffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdea00b4ac0 0x7fdea006cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fdea4db1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 0x7fdea014fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fde9ffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdea00b4ac0 0x7fdea006cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39322/0 (socket says 172.21.15.121:39322) 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fde9ffff640 1 -- 172.21.15.121:0/2387752745 learned_addr learned my addr 172.21.15.121:0/2387752745 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fdea55b2640 1 --2- 172.21.15.121:0/2387752745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdea015f6a0 0x7fdea00c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fde9ffff640 1 -- 172.21.15.121:0/2387752745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 msgr2=0x7fdea014fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fde9ffff640 1 --2- 172.21.15.121:0/2387752745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 0x7fdea014fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fde9ffff640 1 -- 172.21.15.121:0/2387752745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdea015f6a0 msgr2=0x7fdea00c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fde9ffff640 1 --2- 172.21.15.121:0/2387752745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdea015f6a0 0x7fdea00c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.233+0000 7fde9ffff640 1 -- 172.21.15.121:0/2387752745 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea006d5b0 con 0x7fdea00b4ac0 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.234+0000 7fdea4db1640 1 --2- 172.21.15.121:0/2387752745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 0x7fdea014fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.234+0000 7fde9ffff640 1 --2- 172.21.15.121:0/2387752745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdea00b4ac0 0x7fdea006cbd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00e9b0 tx=0x7fde8c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.234+0000 7fde9d7fa640 1 -- 172.21.15.121:0/2387752745 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c017070 con 0x7fdea00b4ac0 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.234+0000 7fde9d7fa640 1 -- 172.21.15.121:0/2387752745 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde8c00c9a0 con 0x7fdea00b4ac0 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.234+0000 7fde9d7fa640 1 -- 172.21.15.121:0/2387752745 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c00cd90 con 0x7fdea00b4ac0 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.234+0000 7fdea7015640 1 -- 172.21.15.121:0/2387752745 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdea006d890 con 0x7fdea00b4ac0 2024-04-25T01:21:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.234+0000 7fdea7015640 1 -- 172.21.15.121:0/2387752745 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdea01501e0 con 0x7fdea00b4ac0 2024-04-25T01:21:23.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.234+0000 7fde9d7fa640 1 -- 172.21.15.121:0/2387752745 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fde8c01a070 con 0x7fdea00b4ac0 2024-04-25T01:21:23.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.234+0000 7fde9d7fa640 1 -- 172.21.15.121:0/2387752745 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fde8c01cba0 con 0x7fdea00b4ac0 2024-04-25T01:21:23.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.235+0000 7fdea7015640 1 -- 172.21.15.121:0/2387752745 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdea0068180 con 0x7fdea00b4ac0 2024-04-25T01:21:23.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.237+0000 7fde9d7fa640 1 -- 172.21.15.121:0/2387752745 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fde8c01c600 con 0x7fdea00b4ac0 2024-04-25T01:21:23.270 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:23.289+0000 7f4756320040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:21:23.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.310+0000 7fdea7015640 1 -- 172.21.15.121:0/2387752745 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "79293f60-29fe-4279-90a3-5d95cd553b00", "id": 7} v 0) -- 0x7fdea00b4010 con 0x7fdea00b4ac0 2024-04-25T01:21:23.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.318+0000 7fde9d7fa640 1 -- 172.21.15.121:0/2387752745 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "79293f60-29fe-4279-90a3-5d95cd553b00", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fde8c01c910 con 0x7fdea00b4ac0 2024-04-25T01:21:23.300 INFO:teuthology.orchestra.run.smithi121.stdout:7 2024-04-25T01:21:23.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.319+0000 7fdea7015640 1 -- 172.21.15.121:0/2387752745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdea00b4ac0 msgr2=0x7fdea006cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:23.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.319+0000 7fdea7015640 1 --2- 172.21.15.121:0/2387752745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdea00b4ac0 0x7fdea006cbd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00e9b0 tx=0x7fde8c00ee70 comp rx=0 tx=0).stop 2024-04-25T01:21:23.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.319+0000 7fdea7015640 1 -- 172.21.15.121:0/2387752745 shutdown_connections 2024-04-25T01:21:23.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.319+0000 7fdea7015640 1 --2- 172.21.15.121:0/2387752745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdea015f6a0 0x7fdea00c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.319+0000 7fdea7015640 1 --2- 172.21.15.121:0/2387752745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdea00b4ac0 0x7fdea006cbd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.319+0000 7fdea7015640 1 --2- 172.21.15.121:0/2387752745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdea00b2be0 0x7fdea014fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.319+0000 7fdea7015640 1 -- 172.21.15.121:0/2387752745 >> 172.21.15.121:0/2387752745 conn(0x7fdea00b37a0 msgr2=0x7fdea00ce900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:23.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.319+0000 7fdea7015640 1 -- 172.21.15.121:0/2387752745 shutdown_connections 2024-04-25T01:21:23.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:21:23.319+0000 7fdea7015640 1 -- 172.21.15.121:0/2387752745 wait complete. 2024-04-25T01:21:23.311 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T01:21:23.311 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-25T01:21:23.313 INFO:tasks.ceph.osd.0:Started 2024-04-25T01:21:23.314 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T01:21:23.314 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-25T01:21:23.316 INFO:tasks.ceph.osd.1:Started 2024-04-25T01:21:23.316 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T01:21:23.316 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-25T01:21:23.318 INFO:tasks.ceph.osd.2:Started 2024-04-25T01:21:23.318 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T01:21:23.318 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-25T01:21:23.321 INFO:tasks.ceph.osd.3:Started 2024-04-25T01:21:23.321 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-25T01:21:23.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-25T01:21:23.323 INFO:tasks.ceph.osd.4:Started 2024-04-25T01:21:23.323 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-25T01:21:23.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-25T01:21:23.325 INFO:tasks.ceph.osd.5:Started 2024-04-25T01:21:23.325 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-25T01:21:23.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-25T01:21:23.330 INFO:tasks.ceph.osd.6:Started 2024-04-25T01:21:23.331 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-25T01:21:23.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-25T01:21:23.335 INFO:tasks.ceph.osd.7:Started 2024-04-25T01:21:23.335 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:21:23.427 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:23.445+0000 7fad55410040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:21:23.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.473+0000 7f028a31d640 1 Processor -- start 2024-04-25T01:21:23.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.474+0000 7f028a31d640 1 -- start start 2024-04-25T01:21:23.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.474+0000 7f028a31d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 0x7f02840ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:23.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.474+0000 7f028a31d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f028406fa50 0x7f028406feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:23.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.474+0000 7f028a31d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02840cee20 con 0x7f0284161e40 2024-04-25T01:21:23.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.474+0000 7f028a31d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f02840d0f40 con 0x7f028406fa50 2024-04-25T01:21:23.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.474+0000 7f028a31d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f02840d0330 con 0x7f02840ce500 2024-04-25T01:21:23.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.475+0000 7f0283fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 0x7f02840ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:23.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.475+0000 7f02837fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f028406fa50 0x7f028406feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:23.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.475+0000 7f02888ba640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0284161e40 0x7f02841622d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:37846/0 (socket says 172.21.15.77:37846) 2024-04-25T01:21:23.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.475+0000 7f0283fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 0x7f02840ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:58868/0 (socket says 172.21.15.77:58868) 2024-04-25T01:21:23.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.475+0000 7f02888ba640 1 -- 172.21.15.77:0/3978683388 learned_addr learned my addr 172.21.15.77:0/3978683388 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:23.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.475+0000 7f0283fff640 1 -- 172.21.15.77:0/3978683388 >> v2:172.21.15.121:3300/0 conn(0x7f028406fa50 msgr2=0x7f028406feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:23.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.475+0000 7f0283fff640 1 --2- 172.21.15.77:0/3978683388 >> v2:172.21.15.121:3300/0 conn(0x7f028406fa50 0x7f028406feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.475+0000 7f0283fff640 1 -- 172.21.15.77:0/3978683388 >> v1:172.21.15.121:6789/0 conn(0x7f0284161e40 legacy=0x7f02841622d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:23.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.475+0000 7f0283fff640 1 -- 172.21.15.77:0/3978683388 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02840d0610 con 0x7f02840ce500 2024-04-25T01:21:23.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f0283fff640 1 --2- 172.21.15.77:0/3978683388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 0x7f02840ce8e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f02780097d0 tx=0x7f027802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd6038f03af6fa96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:23.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f0282ffd640 1 -- 172.21.15.77:0/3978683388 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278037070 con 0x7f02840ce500 2024-04-25T01:21:23.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f0282ffd640 1 -- 172.21.15.77:0/3978683388 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0278032440 con 0x7f02840ce500 2024-04-25T01:21:23.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f0282ffd640 1 -- 172.21.15.77:0/3978683388 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278032810 con 0x7f02840ce500 2024-04-25T01:21:23.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f028a31d640 1 -- 172.21.15.77:0/3978683388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 msgr2=0x7f02840ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:23.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f028a31d640 1 --2- 172.21.15.77:0/3978683388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 0x7f02840ce8e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f02780097d0 tx=0x7f027802a1c0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f028a31d640 1 -- 172.21.15.77:0/3978683388 shutdown_connections 2024-04-25T01:21:23.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f028a31d640 1 --2- 172.21.15.77:0/3978683388 >> v2:172.21.15.121:3300/0 conn(0x7f028406fa50 0x7f028406feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f028a31d640 1 --2- 172.21.15.77:0/3978683388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 0x7f02840ce8e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f028a31d640 1 -- 172.21.15.77:0/3978683388 >> 172.21.15.77:0/3978683388 conn(0x7f02840d1c80 msgr2=0x7f02840d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:23.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f028a31d640 1 -- 172.21.15.77:0/3978683388 shutdown_connections 2024-04-25T01:21:23.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.476+0000 7f028a31d640 1 -- 172.21.15.77:0/3978683388 wait complete. 2024-04-25T01:21:23.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.478+0000 7f028a31d640 1 Processor -- start 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.478+0000 7f028a31d640 1 -- start start 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.478+0000 7f028a31d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f028406fa50 0x7f02840c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.478+0000 7f028a31d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 0x7f02841f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.478+0000 7f028a31d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0284161e40 0x7f0284146f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.478+0000 7f028a31d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f02840b1390 con 0x7f0284161e40 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.478+0000 7f028a31d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f02840d0bd0 con 0x7f028406fa50 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f028a31d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f02840b1510 con 0x7f02840ce500 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f02888ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0284161e40 0x7f0284146f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f02888ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0284161e40 0x7f0284146f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45878/0 (socket says 172.21.15.77:45878) 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f02888ba640 1 -- 172.21.15.77:0/2796045566 learned_addr learned my addr 172.21.15.77:0/2796045566 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0283fff640 1 --2- 172.21.15.77:0/2796045566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f028406fa50 0x7f02840c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f02837fe640 1 --2- 172.21.15.77:0/2796045566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 0x7f02841f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0283fff640 1 -- 172.21.15.77:0/2796045566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 msgr2=0x7f02841f4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:23.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0283fff640 1 --2- 172.21.15.77:0/2796045566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 0x7f02841f4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0283fff640 1 -- 172.21.15.77:0/2796045566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0284161e40 msgr2=0x7f0284146f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0283fff640 1 --2- 172.21.15.77:0/2796045566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0284161e40 0x7f0284146f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0283fff640 1 -- 172.21.15.77:0/2796045566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02841f50e0 con 0x7f028406fa50 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f02837fe640 1 --2- 172.21.15.77:0/2796045566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 0x7f02841f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0283fff640 1 --2- 172.21.15.77:0/2796045566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f028406fa50 0x7f02840c7770 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0278031000 tx=0x7f02780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0280ff9640 1 -- 172.21.15.77:0/2796045566 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278041030 con 0x7f028406fa50 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0280ff9640 1 -- 172.21.15.77:0/2796045566 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f027803d420 con 0x7f028406fa50 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f028a31d640 1 -- 172.21.15.77:0/2796045566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02840c6400 con 0x7f028406fa50 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0280ff9640 1 -- 172.21.15.77:0/2796045566 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278037040 con 0x7f028406fa50 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f028a31d640 1 -- 172.21.15.77:0/2796045566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02840c69b0 con 0x7f028406fa50 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0280ff9640 1 -- 172.21.15.77:0/2796045566 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0278044070 con 0x7f028406fa50 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.479+0000 7f0280ff9640 1 -- 172.21.15.77:0/2796045566 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f027803b3f0 con 0x7f028406fa50 2024-04-25T01:21:23.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.480+0000 7f028a31d640 1 -- 172.21.15.77:0/2796045566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0284068180 con 0x7f028406fa50 2024-04-25T01:21:23.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.482+0000 7f0280ff9640 1 -- 172.21.15.77:0/2796045566 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f02841f50e0 con 0x7f028406fa50 2024-04-25T01:21:23.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.543+0000 7f028a31d640 1 -- 172.21.15.77:0/2796045566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02840b7c00 con 0x7f028406fa50 2024-04-25T01:21:23.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.543+0000 7f0280ff9640 1 -- 172.21.15.77:0/2796045566 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f027803d7f0 con 0x7f028406fa50 2024-04-25T01:21:23.524 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:23.524 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:23.312611+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:21:23.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.544+0000 7f028a31d640 1 -- 172.21.15.77:0/2796045566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f028406fa50 msgr2=0x7f02840c7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.544+0000 7f028a31d640 1 --2- 172.21.15.77:0/2796045566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f028406fa50 0x7f02840c7770 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0278031000 tx=0x7f02780046c0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.544+0000 7f028a31d640 1 -- 172.21.15.77:0/2796045566 shutdown_connections 2024-04-25T01:21:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.544+0000 7f028a31d640 1 --2- 172.21.15.77:0/2796045566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0284161e40 0x7f0284146f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.544+0000 7f028a31d640 1 --2- 172.21.15.77:0/2796045566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02840ce500 0x7f02841f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.544+0000 7f028a31d640 1 --2- 172.21.15.77:0/2796045566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f028406fa50 0x7f02840c7770 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.544+0000 7f028a31d640 1 -- 172.21.15.77:0/2796045566 >> 172.21.15.77:0/2796045566 conn(0x7f02840d1c80 msgr2=0x7f02840afab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.544+0000 7f028a31d640 1 -- 172.21.15.77:0/2796045566 shutdown_connections 2024-04-25T01:21:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:23.544+0000 7f028a31d640 1 -- 172.21.15.77:0/2796045566 wait complete. 2024-04-25T01:21:23.536 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T01:21:23.536 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T01:21:23.569 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:23.588+0000 7f4756320040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:21:23.621 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:23.640+0000 7f4756320040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:21:23.688 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:23.707+0000 7fad55410040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:21:23.738 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:23.757+0000 7f4756320040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:21:23.741 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:23.759+0000 7fad55410040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:21:23.858 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:23.876+0000 7fad55410040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:21:23.925 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-04-25T01:21:23.944+0000 7f4762815640 -1 Falling back to public interface 2024-04-25T01:21:23.925 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-04-25T01:21:23.944+0000 7f8bb4357640 -1 Falling back to public interface 2024-04-25T01:21:23.925 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-04-25T01:21:23.944+0000 7f29a4e86640 -1 Falling back to public interface 2024-04-25T01:21:23.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-04-25T01:21:23.945+0000 7f8bb3306640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6808/0: (98) Address already in use 2024-04-25T01:21:23.929 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-25T01:21:23.947+0000 7fc62034e640 -1 Falling back to public interface 2024-04-25T01:21:23.929 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-25T01:21:23.947+0000 7fa396c1c640 -1 Falling back to public interface 2024-04-25T01:21:23.929 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-25T01:21:23.947+0000 7fa3955b9640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2024-04-25T01:21:23.933 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-04-25T01:21:23.952+0000 7fdd27f14640 -1 Falling back to public interface 2024-04-25T01:21:23.949 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-25T01:21:23.967+0000 7f6ca5a86640 -1 Falling back to public interface 2024-04-25T01:21:23.949 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-25T01:21:23.967+0000 7fefbf0f0640 -1 Falling back to public interface 2024-04-25T01:21:23.949 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-25T01:21:23.968+0000 7f6ca4a35640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2024-04-25T01:21:24.020 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:24.039+0000 7f4756320040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:21:24.068 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:24.087+0000 7f4756320040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:21:24.142 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:24.160+0000 7fad55410040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:21:24.190 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:24.208+0000 7fad55410040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:21:24.226 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:24.245+0000 7f4756320040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:21:24.342 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T01:21:24.351 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:24.366+0000 7fad55410040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:21:24.362 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:24.382+0000 7f4756320040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:21:24.414 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:24.433+0000 7f4756320040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:21:24.433 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.452+0000 7f2ad8f29640 1 Processor -- start 2024-04-25T01:21:24.433 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.452+0000 7f2ad8f29640 1 -- start start 2024-04-25T01:21:24.435 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad8f29640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2ad4071c30 0x7f2ad406d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:24.435 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad8f29640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ad415bf50 0x7f2ad40cc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:24.435 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad8f29640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ad406def0 con 0x7f2ad4072010 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad8f29640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2ad40cfe70 con 0x7f2ad4071c30 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad8f29640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad40cb950 con 0x7f2ad415bf50 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad1d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2ad4072010 0x7f2ad40ceb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49314/0 (socket says 172.21.15.77:49314) 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad1d9b640 1 -- 172.21.15.77:0/1892874594 learned_addr learned my addr 172.21.15.77:0/1892874594 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad2d9d640 1 --2- 172.21.15.77:0/1892874594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ad415bf50 0x7f2ad40cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad259c640 1 --2- 172.21.15.77:0/1892874594 >> v2:172.21.15.121:3300/0 conn(0x7f2ad4071c30 0x7f2ad406d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad159a640 1 -- 172.21.15.77:0/1892874594 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 715259712 0 0) 0x7f2ad406def0 con 0x7f2ad4072010 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad159a640 1 -- 172.21.15.77:0/1892874594 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2abc005730 con 0x7f2ad4072010 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad259c640 1 -- 172.21.15.77:0/1892874594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ad415bf50 msgr2=0x7f2ad40cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad259c640 1 --2- 172.21.15.77:0/1892874594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ad415bf50 0x7f2ad40cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad259c640 1 -- 172.21.15.77:0/1892874594 >> v1:172.21.15.77:6789/0 conn(0x7f2ad4072010 legacy=0x7f2ad40ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad159a640 1 -- 172.21.15.77:0/1892874594 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1130744602 0 0) 0x7f2abc005730 con 0x7f2ad4072010 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad259c640 1 -- 172.21.15.77:0/1892874594 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad40cf5d0 con 0x7f2ad4071c30 2024-04-25T01:21:24.436 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.453+0000 7f2ad2d9d640 1 --2- 172.21.15.77:0/1892874594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ad415bf50 0x7f2ad40cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:24.437 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.454+0000 7f2ad259c640 1 --2- 172.21.15.77:0/1892874594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ad4071c30 0x7f2ad406d9b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40097a0 tx=0x7f2ac402a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f6c870d1e7c5ae2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:24.437 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.454+0000 7f2ad159a640 1 -- 172.21.15.77:0/1892874594 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4037070 con 0x7f2ad4071c30 2024-04-25T01:21:24.437 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.454+0000 7f2ad159a640 1 -- 172.21.15.77:0/1892874594 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ac402aba0 con 0x7f2ad4071c30 2024-04-25T01:21:24.437 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.454+0000 7f2ad159a640 1 -- 172.21.15.77:0/1892874594 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8001670 con 0x7f2ad4071c30 2024-04-25T01:21:24.440 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.455+0000 7f2ad8f29640 1 -- 172.21.15.77:0/1892874594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ad4071c30 msgr2=0x7f2ad406d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:24.440 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.455+0000 7f2ad8f29640 1 --2- 172.21.15.77:0/1892874594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ad4071c30 0x7f2ad406d9b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40097a0 tx=0x7f2ac402a110 comp rx=0 tx=0).stop 2024-04-25T01:21:24.440 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.455+0000 7f2ad8f29640 1 -- 172.21.15.77:0/1892874594 shutdown_connections 2024-04-25T01:21:24.440 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.455+0000 7f2ad8f29640 1 --2- 172.21.15.77:0/1892874594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ad415bf50 0x7f2ad40cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:24.440 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.455+0000 7f2ad8f29640 1 --2- 172.21.15.77:0/1892874594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ad4071c30 0x7f2ad406d9b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.455+0000 7f2ad8f29640 1 -- 172.21.15.77:0/1892874594 >> 172.21.15.77:0/1892874594 conn(0x7f2ad40d0ba0 msgr2=0x7f2ad40d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.455+0000 7f2ad8f29640 1 -- 172.21.15.77:0/1892874594 shutdown_connections 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.455+0000 7f2ad8f29640 1 -- 172.21.15.77:0/1892874594 wait complete. 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.456+0000 7f2ad8f29640 1 Processor -- start 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.456+0000 7f2ad8f29640 1 -- start start 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad8f29640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ad4071c30 0x7f2ad40ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad8f29640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad4072010 0x7f2ad40c5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad8f29640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ad415bf50 0x7f2ad414ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad8f29640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad40cbbf0 con 0x7f2ad4072010 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad8f29640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad40cfb90 con 0x7f2ad4071c30 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad8f29640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad415eb00 con 0x7f2ad415bf50 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad4072010 0x7f2ad40c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:24.441 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad4072010 0x7f2ad40c5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45880/0 (socket says 172.21.15.77:45880) 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad1d9b640 1 -- 172.21.15.77:0/2544608040 learned_addr learned my addr 172.21.15.77:0/2544608040 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad259c640 1 --2- 172.21.15.77:0/2544608040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ad4071c30 0x7f2ad40ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad2d9d640 1 --2- 172.21.15.77:0/2544608040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ad415bf50 0x7f2ad414ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad1d9b640 1 -- 172.21.15.77:0/2544608040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ad415bf50 msgr2=0x7f2ad414ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad1d9b640 1 --2- 172.21.15.77:0/2544608040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ad415bf50 0x7f2ad414ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad1d9b640 1 -- 172.21.15.77:0/2544608040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ad4071c30 msgr2=0x7f2ad40ca780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad1d9b640 1 --2- 172.21.15.77:0/2544608040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ad4071c30 0x7f2ad40ca780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad1d9b640 1 -- 172.21.15.77:0/2544608040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad40c55d0 con 0x7f2ad4072010 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.457+0000 7f2ad259c640 1 --2- 172.21.15.77:0/2544608040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ad4071c30 0x7f2ad40ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.458+0000 7f2ad1d9b640 1 --2- 172.21.15.77:0/2544608040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad4072010 0x7f2ad40c5090 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800c780 tx=0x7f2ac800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.458+0000 7f2ac2ffd640 1 -- 172.21.15.77:0/2544608040 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8017070 con 0x7f2ad4072010 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.458+0000 7f2ac2ffd640 1 -- 172.21.15.77:0/2544608040 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ac8005ad0 con 0x7f2ad4072010 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.458+0000 7f2ad8f29640 1 -- 172.21.15.77:0/2544608040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad40c4900 con 0x7f2ad4072010 2024-04-25T01:21:24.442 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.458+0000 7f2ac2ffd640 1 -- 172.21.15.77:0/2544608040 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8007070 con 0x7f2ad4072010 2024-04-25T01:21:24.443 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.458+0000 7f2ad8f29640 1 -- 172.21.15.77:0/2544608040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad4150500 con 0x7f2ad4072010 2024-04-25T01:21:24.443 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.458+0000 7f2ac2ffd640 1 -- 172.21.15.77:0/2544608040 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2ac801a070 con 0x7f2ad4072010 2024-04-25T01:21:24.443 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.458+0000 7f2ac2ffd640 1 -- 172.21.15.77:0/2544608040 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f2ac8012ad0 con 0x7f2ad4072010 2024-04-25T01:21:24.443 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.459+0000 7f2ad8f29640 1 -- 172.21.15.77:0/2544608040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad404f500 con 0x7f2ad4072010 2024-04-25T01:21:24.443 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.461+0000 7f2ac2ffd640 1 -- 172.21.15.77:0/2544608040 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2ac8021020 con 0x7f2ad4072010 2024-04-25T01:21:24.483 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:24.501+0000 7fad55410040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:21:24.488 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:24.507+0000 7f4756320040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:21:24.503 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.522+0000 7f2ad8f29640 1 -- 172.21.15.77:0/2544608040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ad4072760 con 0x7f2ad4072010 2024-04-25T01:21:24.503 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.522+0000 7f2ac2ffd640 1 -- 172.21.15.77:0/2544608040 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f2ac800a640 con 0x7f2ad4072010 2024-04-25T01:21:24.503 INFO:teuthology.misc.health.smithi077.stdout: 2024-04-25T01:21:24.504 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":10,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:23.312611+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:21:24.504 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.523+0000 7f2ad8f29640 1 -- 172.21.15.77:0/2544608040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad4072010 msgr2=0x7f2ad40c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:24.505 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.523+0000 7f2ad8f29640 1 --2- 172.21.15.77:0/2544608040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad4072010 0x7f2ad40c5090 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800c780 tx=0x7f2ac800cb80 comp rx=0 tx=0).stop 2024-04-25T01:21:24.505 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.523+0000 7f2ad8f29640 1 -- 172.21.15.77:0/2544608040 shutdown_connections 2024-04-25T01:21:24.505 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.523+0000 7f2ad8f29640 1 --2- 172.21.15.77:0/2544608040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad4072010 0x7f2ad40c5090 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:24.505 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.523+0000 7f2ad8f29640 1 --2- 172.21.15.77:0/2544608040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ad415bf50 0x7f2ad414ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:24.505 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.523+0000 7f2ad8f29640 1 --2- 172.21.15.77:0/2544608040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ad4071c30 0x7f2ad40ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:24.505 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.523+0000 7f2ad8f29640 1 -- 172.21.15.77:0/2544608040 >> 172.21.15.77:0/2544608040 conn(0x7f2ad40d0ba0 msgr2=0x7f2ad406ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:24.505 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.523+0000 7f2ad8f29640 1 -- 172.21.15.77:0/2544608040 shutdown_connections 2024-04-25T01:21:24.505 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:24.523+0000 7f2ad8f29640 1 -- 172.21.15.77:0/2544608040 wait complete. 2024-04-25T01:21:24.514 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T01:21:24.534 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:24.552+0000 7fad55410040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:21:24.535 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:24.555+0000 7f4756320040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:21:24.603 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:24.621+0000 7fad55410040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:21:24.648 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:24.666+0000 7fad55410040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:21:25.009 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:25.027+0000 7f4756320040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:21:25.125 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:25.143+0000 7fad55410040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:21:25.126 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:25.145+0000 7f4756320040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:21:25.244 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-25T01:21:25.263+0000 7fad55410040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:21:26.144 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:26.163+0000 7f46f6e2f640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T01:21:26.144 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-04-25T01:21:26.163+0000 7f46f6e2f640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T01:21:27.412 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-04-25T01:21:27.431+0000 7f4762815640 -1 osd.2 0 log_to_monitors true 2024-04-25T01:21:27.446 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-25T01:21:27.464+0000 7fefbf0f0640 -1 osd.6 0 log_to_monitors true 2024-04-25T01:21:27.453 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-25T01:21:27.472+0000 7fa396c1c640 -1 osd.4 0 log_to_monitors true 2024-04-25T01:21:27.454 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-25T01:21:27.472+0000 7fc62034e640 -1 osd.7 0 log_to_monitors true 2024-04-25T01:21:27.456 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-25T01:21:27.475+0000 7f6ca5a86640 -1 osd.5 0 log_to_monitors true 2024-04-25T01:21:27.459 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-04-25T01:21:27.478+0000 7f29a4e86640 -1 osd.1 0 log_to_monitors true 2024-04-25T01:21:27.477 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-04-25T01:21:27.496+0000 7f8bb4357640 -1 osd.3 0 log_to_monitors true 2024-04-25T01:21:27.483 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-04-25T01:21:27.502+0000 7fdd27f14640 -1 osd.0 0 log_to_monitors true 2024-04-25T01:21:31.216 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-25T01:21:31.234+0000 7fc61c2e4640 -1 osd.7 0 waiting for initial osdmap 2024-04-25T01:21:31.229 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-25T01:21:31.247+0000 7fc614104640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:31.262 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-25T01:21:31.280+0000 7fa392db4640 -1 osd.4 0 waiting for initial osdmap 2024-04-25T01:21:31.262 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-25T01:21:31.280+0000 7fefbb086640 -1 osd.6 0 waiting for initial osdmap 2024-04-25T01:21:31.263 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-25T01:21:31.281+0000 7f6ca1a1c640 -1 osd.5 0 waiting for initial osdmap 2024-04-25T01:21:31.276 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-25T01:21:31.295+0000 7fefb2ea6640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:31.279 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-04-25T01:21:31.298+0000 7f8bb02ed640 -1 osd.3 0 waiting for initial osdmap 2024-04-25T01:21:31.281 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-25T01:21:31.300+0000 7f6c9983c640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:31.283 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-25T01:21:31.302+0000 7fa38a3c0640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:31.297 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-04-25T01:21:31.316+0000 7f8ba810d640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:31.304 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-04-25T01:21:31.323+0000 7fdd23eaa640 -1 osd.0 0 waiting for initial osdmap 2024-04-25T01:21:31.319 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T01:21:31.320 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-04-25T01:21:31.338+0000 7f29a0e1c640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T01:21:31.321 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-04-25T01:21:31.339+0000 7fdd1bcca640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:31.330 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-04-25T01:21:31.349+0000 7f2998c3c640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:31.335 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-04-25T01:21:31.354+0000 7f475e0fb640 -1 osd.2 0 waiting for initial osdmap 2024-04-25T01:21:31.348 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-04-25T01:21:31.368+0000 7f4755f1b640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:31.409 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.428+0000 7f7c6b3ee640 1 Processor -- start 2024-04-25T01:21:31.409 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.429+0000 7f7c6b3ee640 1 -- start start 2024-04-25T01:21:31.414 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.429+0000 7f7c6b3ee640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c6415e120 0x7f7c640d23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:31.414 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.429+0000 7f7c6b3ee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640ca540 0x7f7c640d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:31.414 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.429+0000 7f7c6b3ee640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c640d21e0 con 0x7f7c6415e500 2024-04-25T01:21:31.414 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.429+0000 7f7c6b3ee640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7c640c7a90 con 0x7f7c6415e120 2024-04-25T01:21:31.414 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.429+0000 7f7c6b3ee640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7c640d1f00 con 0x7f7c640ca540 2024-04-25T01:21:31.414 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c68989640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7c6415e500 0x7f7c640cea20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:42006/0 (socket says 172.21.15.77:42006) 2024-04-25T01:21:31.414 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c68989640 1 -- 172.21.15.77:0/1277990459 learned_addr learned my addr 172.21.15.77:0/1277990459 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:31.414 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c6998b640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640ca540 0x7f7c640d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:31.414 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c6918a640 1 --2- 172.21.15.77:0/1277990459 >> v2:172.21.15.121:3300/0 conn(0x7f7c6415e120 0x7f7c640d23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:31.414 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840243400 0 0) 0x7f7c640d21e0 con 0x7f7c6415e500 2024-04-25T01:21:31.414 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c50003520 con 0x7f7c6415e500 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1138969853 0 0) 0x7f7c50003520 con 0x7f7c6415e500 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640ca540 msgr2=0x7f7c640d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640ca540 0x7f7c640d2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 >> v2:172.21.15.121:3300/0 conn(0x7f7c6415e120 msgr2=0x7f7c640d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 --2- 172.21.15.77:0/1277990459 >> v2:172.21.15.121:3300/0 conn(0x7f7c6415e120 0x7f7c640d23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c640c7d30 con 0x7f7c6415e500 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f7c60001670 con 0x7f7c6415e500 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 >> v1:172.21.15.77:6789/0 conn(0x7f7c6415e500 legacy=0x7f7c640cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 0x7f7c50003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640ca540 0x7f7c50004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c50004d40 0x7f7c500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c640c7a90 con 0x7f7c50003a20 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7c640d1f00 con 0x7f7c50004d40 2024-04-25T01:21:31.415 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7c5000ba10 con 0x7f7c640ca540 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c6918a640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 0x7f7c50003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c68989640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640ca540 0x7f7c50004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c6998b640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c50004d40 0x7f7c500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c6918a640 1 -- 172.21.15.77:0/1277990459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640ca540 msgr2=0x7f7c50004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c6918a640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640ca540 0x7f7c50004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c6918a640 1 -- 172.21.15.77:0/1277990459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c50004d40 msgr2=0x7f7c500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c6918a640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c50004d40 0x7f7c500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.430+0000 7f7c6918a640 1 -- 172.21.15.77:0/1277990459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c50003520 con 0x7f7c50003a20 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.431+0000 7f7c6918a640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 0x7f7c50003e40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009ac0 tx=0x7f7c50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a892dbd3e0f3c17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.431+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c54036070 con 0x7f7c50003a20 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.431+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c54031400 con 0x7f7c50003a20 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.431+0000 7f7c5bfff640 1 -- 172.21.15.77:0/1277990459 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c54009690 con 0x7f7c50003a20 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.432+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/1277990459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 msgr2=0x7f7c50003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.432+0000 7f7c6b3ee640 1 --2- 172.21.15.77:0/1277990459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 0x7f7c50003e40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009ac0 tx=0x7f7c50005830 comp rx=0 tx=0).stop 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.432+0000 7f7c6998b640 1 -- 172.21.15.77:0/1277990459 reap_dead start 2024-04-25T01:21:31.416 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.432+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/1277990459 shutdown_connections 2024-04-25T01:21:31.417 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.432+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/1277990459 >> 172.21.15.77:0/1277990459 conn(0x7f7c640d3e10 msgr2=0x7f7c640d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:31.417 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.432+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/1277990459 shutdown_connections 2024-04-25T01:21:31.417 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.432+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/1277990459 wait complete. 2024-04-25T01:21:31.417 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.433+0000 7f7c6b3ee640 1 Processor -- start 2024-04-25T01:21:31.417 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.433+0000 7f7c6b3ee640 1 -- start start 2024-04-25T01:21:31.419 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6b3ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 0x7f7c641f4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6b3ee640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c64153230 0x7f7c640b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6b3ee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640b9140 0x7f7c641f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6b3ee640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c640d4930 con 0x7f7c50003a20 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6b3ee640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7c640d1bf0 con 0x7f7c64153230 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6b3ee640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7c640d4ab0 con 0x7f7c640b9140 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6918a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 0x7f7c641f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6918a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 0x7f7c641f4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58282/0 (socket says 172.21.15.77:58282) 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6918a640 1 -- 172.21.15.77:0/620690537 learned_addr learned my addr 172.21.15.77:0/620690537 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6998b640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640b9140 0x7f7c641f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c68989640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c64153230 0x7f7c640b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6918a640 1 -- 172.21.15.77:0/620690537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640b9140 msgr2=0x7f7c641f70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6918a640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640b9140 0x7f7c641f70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:31.420 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6918a640 1 -- 172.21.15.77:0/620690537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c64153230 msgr2=0x7f7c640b8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6918a640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c64153230 0x7f7c640b8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6918a640 1 -- 172.21.15.77:0/620690537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c641f4a50 con 0x7f7c50003a20 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c68989640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c64153230 0x7f7c640b8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6918a640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 0x7f7c641f4510 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009be0 tx=0x7f7c54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c59ffb640 1 -- 172.21.15.77:0/620690537 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c54040020 con 0x7f7c50003a20 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c59ffb640 1 -- 172.21.15.77:0/620690537 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c54003950 con 0x7f7c50003a20 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c59ffb640 1 -- 172.21.15.77:0/620690537 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c54036040 con 0x7f7c50003a20 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/620690537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c640cb210 con 0x7f7c50003a20 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.434+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/620690537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c640cb7d0 con 0x7f7c50003a20 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.435+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/620690537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c6404f500 con 0x7f7c50003a20 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.436+0000 7f7c59ffb640 1 -- 172.21.15.77:0/620690537 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f7c54043020 con 0x7f7c50003a20 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.436+0000 7f7c59ffb640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c4806bdd0 0x7f7c4806e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.436+0000 7f7c68989640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c4806bdd0 0x7f7c4806e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.436+0000 7f7c59ffb640 1 -- 172.21.15.77:0/620690537 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f7c540aa0c0 con 0x7f7c50003a20 2024-04-25T01:21:31.421 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.437+0000 7f7c68989640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c4806bdd0 0x7f7c4806e290 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7c640b98e0 tx=0x7f7c60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:31.422 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.441+0000 7f7c59ffb640 1 -- 172.21.15.77:0/620690537 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7c5404a070 con 0x7f7c50003a20 2024-04-25T01:21:31.599 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.617+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/620690537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c640b9520 con 0x7f7c50003a20 2024-04-25T01:21:31.599 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.618+0000 7f7c59ffb640 1 -- 172.21.15.77:0/620690537 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f7c540798e0 con 0x7f7c50003a20 2024-04-25T01:21:31.599 INFO:teuthology.misc.health.smithi077.stdout: 2024-04-25T01:21:31.599 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":12,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:29.168319+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:21:31.601 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/620690537 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c4806bdd0 msgr2=0x7f7c4806e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:31.601 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c4806bdd0 0x7f7c4806e290 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7c640b98e0 tx=0x7f7c60009040 comp rx=0 tx=0).stop 2024-04-25T01:21:31.601 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/620690537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 msgr2=0x7f7c641f4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:31.601 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 0x7f7c641f4510 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009be0 tx=0x7f7c54004730 comp rx=0 tx=0).stop 2024-04-25T01:21:31.602 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/620690537 shutdown_connections 2024-04-25T01:21:31.602 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c4806bdd0 0x7f7c4806e290 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:31.602 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c640b9140 0x7f7c641f70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:31.602 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c64153230 0x7f7c640b8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:31.602 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 --2- 172.21.15.77:0/620690537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c50003a20 0x7f7c641f4510 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:31.602 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/620690537 >> 172.21.15.77:0/620690537 conn(0x7f7c640d3e10 msgr2=0x7f7c64160d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:31.602 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/620690537 shutdown_connections 2024-04-25T01:21:31.602 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:31.620+0000 7f7c6b3ee640 1 -- 172.21.15.77:0/620690537 wait complete. 2024-04-25T01:21:31.610 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T01:21:38.416 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T01:21:38.515 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.533+0000 7fa2e4ae1640 1 Processor -- start 2024-04-25T01:21:38.515 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.533+0000 7fa2e4ae1640 1 -- start start 2024-04-25T01:21:38.516 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.534+0000 7fa2e4ae1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 0x7fa2e00b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.516 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.534+0000 7fa2e4ae1640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa2e00ce4e0 0x7fa2e00ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.516 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.534+0000 7fa2e4ae1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e015eb00 con 0x7fa2e00b3110 2024-04-25T01:21:38.516 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.534+0000 7fa2e4ae1640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa2e006d840 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.516 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.534+0000 7fa2e4ae1640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e00cc350 con 0x7fa2e00d1580 2024-04-25T01:21:38.517 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.534+0000 7fa2ddd9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 0x7fa2e00b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.517 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.534+0000 7fa2ded9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa2e00ce4e0 0x7fa2e00ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.517 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.535+0000 7fa2de59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa2e00b3110 0x7fa2e006d130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:49902/0 (socket says 172.21.15.77:49902) 2024-04-25T01:21:38.517 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.535+0000 7fa2ddd9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 0x7fa2e00b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:59806/0 (socket says 172.21.15.77:59806) 2024-04-25T01:21:38.517 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.535+0000 7fa2de59c640 1 -- 172.21.15.77:0/3485030226 learned_addr learned my addr 172.21.15.77:0/3485030226 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:38.518 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.535+0000 7fa2ded9d640 1 -- 172.21.15.77:0/3485030226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 msgr2=0x7fa2e00b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:38.518 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.535+0000 7fa2ded9d640 1 --2- 172.21.15.77:0/3485030226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 0x7fa2e00b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.518 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.535+0000 7fa2ded9d640 1 -- 172.21.15.77:0/3485030226 >> v1:172.21.15.121:6789/0 conn(0x7fa2e00b3110 legacy=0x7fa2e006d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:38.518 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.535+0000 7fa2ded9d640 1 -- 172.21.15.77:0/3485030226 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e00cc630 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.518 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.535+0000 7fa2ddd9b640 1 --2- 172.21.15.77:0/3485030226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 0x7fa2e00b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:38.518 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2ded9d640 1 --2- 172.21.15.77:0/3485030226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e00ce4e0 0x7fa2e00ce970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009d90 tx=0x7fa2d802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b68c3fae89264e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:38.519 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2dd59a640 1 -- 172.21.15.77:0/3485030226 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8007740 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.519 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2dd59a640 1 -- 172.21.15.77:0/3485030226 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d80078e0 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.519 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2dd59a640 1 -- 172.21.15.77:0/3485030226 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8007cb0 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.519 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3485030226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e00ce4e0 msgr2=0x7fa2e00ce970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:38.519 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2e4ae1640 1 --2- 172.21.15.77:0/3485030226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e00ce4e0 0x7fa2e00ce970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009d90 tx=0x7fa2d802a660 comp rx=0 tx=0).stop 2024-04-25T01:21:38.519 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3485030226 shutdown_connections 2024-04-25T01:21:38.519 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2e4ae1640 1 --2- 172.21.15.77:0/3485030226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e00ce4e0 0x7fa2e00ce970 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.520 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2e4ae1640 1 --2- 172.21.15.77:0/3485030226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 0x7fa2e00b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.520 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3485030226 >> 172.21.15.77:0/3485030226 conn(0x7fa2e00d1c60 msgr2=0x7fa2e00d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:38.520 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3485030226 shutdown_connections 2024-04-25T01:21:38.520 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.536+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3485030226 wait complete. 2024-04-25T01:21:38.522 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.538+0000 7fa2e4ae1640 1 Processor -- start 2024-04-25T01:21:38.522 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.538+0000 7fa2e4ae1640 1 -- start start 2024-04-25T01:21:38.522 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.538+0000 7fa2e4ae1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e00b3110 0x7fa2e0157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.522 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.538+0000 7fa2e4ae1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2e00ce4e0 0x7fa2e01e8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.522 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.538+0000 7fa2e4ae1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 0x7fa2e015b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.522 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.538+0000 7fa2e4ae1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e00cced0 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.522 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.538+0000 7fa2e4ae1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e00ccbf0 con 0x7fa2e00b3110 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.538+0000 7fa2e4ae1640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e00b34f0 con 0x7fa2e00d1580 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ddd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2e00ce4e0 0x7fa2e01e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ddd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2e00ce4e0 0x7fa2e01e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52942/0 (socket says 172.21.15.77:52942) 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ddd9b640 1 -- 172.21.15.77:0/3669825365 learned_addr learned my addr 172.21.15.77:0/3669825365 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ded9d640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 0x7fa2e015b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2de59c640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e00b3110 0x7fa2e0157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ddd9b640 1 -- 172.21.15.77:0/3669825365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 msgr2=0x7fa2e015b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ddd9b640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 0x7fa2e015b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ddd9b640 1 -- 172.21.15.77:0/3669825365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e00b3110 msgr2=0x7fa2e0157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ddd9b640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e00b3110 0x7fa2e0157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ddd9b640 1 -- 172.21.15.77:0/3669825365 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e01f3ca0 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ddd9b640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2e00ce4e0 0x7fa2e01e8170 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400ebe0 tx=0x7fa2d400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ceffd640 1 -- 172.21.15.77:0/3669825365 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4017070 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ceffd640 1 -- 172.21.15.77:0/3669825365 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d400cc30 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.523 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3669825365 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e01e6180 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.524 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2ceffd640 1 -- 172.21.15.77:0/3669825365 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4013470 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.524 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.539+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3669825365 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e01e6740 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.524 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.540+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3669825365 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e004f500 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.524 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.541+0000 7fa2ceffd640 1 -- 172.21.15.77:0/3669825365 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa2d401a070 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.524 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.541+0000 7fa2ceffd640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2c006be60 0x7fa2c006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.525 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.541+0000 7fa2de59c640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2c006be60 0x7fa2c006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.525 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.541+0000 7fa2ceffd640 1 -- 172.21.15.77:0/3669825365 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fa2d408a920 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.525 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.541+0000 7fa2de59c640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2c006be60 0x7fa2c006e320 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0007fd0 tx=0x7fa2d0007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:38.527 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.546+0000 7fa2ceffd640 1 -- 172.21.15.77:0/3669825365 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa2d4059060 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.701 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.719+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3669825365 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2e006bf20 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.702 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.720+0000 7fa2ceffd640 1 -- 172.21.15.77:0/3669825365 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12884 (secure 0 0 0) 0x7fa2d405d360 con 0x7fa2e00ce4e0 2024-04-25T01:21:38.702 INFO:teuthology.misc.health.smithi077.stdout: 2024-04-25T01:21:38.703 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":16,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:35.242898+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:21:38.704 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3669825365 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2c006be60 msgr2=0x7fa2c006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:38.704 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2c006be60 0x7fa2c006e320 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0007fd0 tx=0x7fa2d0007e80 comp rx=0 tx=0).stop 2024-04-25T01:21:38.704 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3669825365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2e00ce4e0 msgr2=0x7fa2e01e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:38.705 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2e00ce4e0 0x7fa2e01e8170 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400ebe0 tx=0x7fa2d400c520 comp rx=0 tx=0).stop 2024-04-25T01:21:38.705 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3669825365 shutdown_connections 2024-04-25T01:21:38.705 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2c006be60 0x7fa2c006e320 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.705 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2e00ce4e0 0x7fa2e01e8170 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.705 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e00d1580 0x7fa2e015b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.705 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 --2- 172.21.15.77:0/3669825365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e00b3110 0x7fa2e0157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.705 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3669825365 >> 172.21.15.77:0/3669825365 conn(0x7fa2e00d1c60 msgr2=0x7fa2e006ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:38.705 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3669825365 shutdown_connections 2024-04-25T01:21:38.705 INFO:teuthology.misc.health.smithi077.stderr:2024-04-25T01:21:38.722+0000 7fa2e4ae1640 1 -- 172.21.15.77:0/3669825365 wait complete. 2024-04-25T01:21:38.715 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-25T01:21:38.715 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T01:21:38.716 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T01:21:38.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T01:21:38.719 INFO:tasks.ceph.mds.a:Started 2024-04-25T01:21:38.719 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T01:21:38.719 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T01:21:38.721 INFO:tasks.ceph.mds.c:Started 2024-04-25T01:21:38.721 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T01:21:38.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T01:21:38.724 INFO:tasks.ceph.mds.b:Started 2024-04-25T01:21:38.724 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T01:21:38.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T01:21:38.727 INFO:tasks.ceph.mds.d:Started 2024-04-25T01:21:38.727 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-25T01:21:38.727 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:21:38.792 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-04-25T01:21:38.792 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-04-25T01:21:38.807 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-25T01:21:38.808 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-25T01:21:38.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.845+0000 7f88eb8e7640 1 Processor -- start 2024-04-25T01:21:38.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.846+0000 7f88eb8e7640 1 -- start start 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.846+0000 7f88eb8e7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e40d4a10 0x7f88e40cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88eb8e7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f88e40b3ee0 0x7f88e40b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88eb8e7640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88e40d1e90 con 0x7f88e415e4c0 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88eb8e7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f88e40b54a0 con 0x7f88e40b3ee0 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88eb8e7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f88e40d2130 con 0x7f88e40d4a10 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88e8e82640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e40d4a10 0x7f88e40cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88e8e82640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e40d4a10 0x7f88e40cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:59852/0 (socket says 172.21.15.77:59852) 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88e9683640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f88e415e4c0 0x7f88e40d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:49920/0 (socket says 172.21.15.77:49920) 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88e8e82640 1 -- 172.21.15.77:0/568857832 learned_addr learned my addr 172.21.15.77:0/568857832 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88e9e84640 1 --2- 172.21.15.77:0/568857832 >> v2:172.21.15.121:3300/0 conn(0x7f88e40b3ee0 0x7f88e40b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88dbfff640 1 -- 172.21.15.77:0/568857832 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2894615072 0 0) 0x7f88e40d1e90 con 0x7f88e415e4c0 2024-04-25T01:21:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88e8e82640 1 -- 172.21.15.77:0/568857832 >> v2:172.21.15.121:3300/0 conn(0x7f88e40b3ee0 msgr2=0x7f88e40b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88e8e82640 1 --2- 172.21.15.77:0/568857832 >> v2:172.21.15.121:3300/0 conn(0x7f88e40b3ee0 0x7f88e40b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88e8e82640 1 -- 172.21.15.77:0/568857832 >> v1:172.21.15.121:6789/0 conn(0x7f88e415e4c0 legacy=0x7f88e40d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.847+0000 7f88e8e82640 1 -- 172.21.15.77:0/568857832 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e40b5780 con 0x7f88e40d4a10 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.848+0000 7f88e8e82640 1 --2- 172.21.15.77:0/568857832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e40d4a10 0x7f88e40cfaf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f88e0009870 tx=0x7f88e002a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e46cffe1ce8913 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.848+0000 7f88dbfff640 1 -- 172.21.15.77:0/568857832 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0037070 con 0x7f88e40d4a10 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.848+0000 7f88dbfff640 1 -- 172.21.15.77:0/568857832 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e0032440 con 0x7f88e40d4a10 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.848+0000 7f88dbfff640 1 -- 172.21.15.77:0/568857832 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0032810 con 0x7f88e40d4a10 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.848+0000 7f88eb8e7640 1 -- 172.21.15.77:0/568857832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e40d4a10 msgr2=0x7f88e40cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.848+0000 7f88eb8e7640 1 --2- 172.21.15.77:0/568857832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e40d4a10 0x7f88e40cfaf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f88e0009870 tx=0x7f88e002a1e0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.848+0000 7f88eb8e7640 1 -- 172.21.15.77:0/568857832 shutdown_connections 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.848+0000 7f88eb8e7640 1 --2- 172.21.15.77:0/568857832 >> v2:172.21.15.121:3300/0 conn(0x7f88e40b3ee0 0x7f88e40b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.848+0000 7f88eb8e7640 1 --2- 172.21.15.77:0/568857832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e40d4a10 0x7f88e40cfaf0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.848+0000 7f88eb8e7640 1 -- 172.21.15.77:0/568857832 >> 172.21.15.77:0/568857832 conn(0x7f88e40d3ef0 msgr2=0x7f88e40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.849+0000 7f88eb8e7640 1 -- 172.21.15.77:0/568857832 shutdown_connections 2024-04-25T01:21:38.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.849+0000 7f88eb8e7640 1 -- 172.21.15.77:0/568857832 wait complete. 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.850+0000 7f88eb8e7640 1 Processor -- start 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.850+0000 7f88eb8e7640 1 -- start start 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88eb8e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88e40b3ee0 0x7f88e41f3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88eb8e7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88e40d4a10 0x7f88e40c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88eb8e7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e415e4c0 0x7f88e40b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88eb8e7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f88e40b30a0 con 0x7f88e40b3ee0 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88eb8e7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f88e40b2f20 con 0x7f88e40d4a10 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e9683640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88e40b3ee0 0x7f88e41f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88eb8e7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f88e415e8a0 con 0x7f88e415e4c0 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e9683640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88e40b3ee0 0x7f88e41f3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53002/0 (socket says 172.21.15.77:53002) 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e9683640 1 -- 172.21.15.77:0/659800391 learned_addr learned my addr 172.21.15.77:0/659800391 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e8e82640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88e40d4a10 0x7f88e40c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e9e84640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e415e4c0 0x7f88e40b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e9683640 1 -- 172.21.15.77:0/659800391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e415e4c0 msgr2=0x7f88e40b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e9683640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e415e4c0 0x7f88e40b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e9683640 1 -- 172.21.15.77:0/659800391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88e40d4a10 msgr2=0x7f88e40c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e9683640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88e40d4a10 0x7f88e40c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e9683640 1 -- 172.21.15.77:0/659800391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e41f3990 con 0x7f88e40b3ee0 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e9e84640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e415e4c0 0x7f88e40b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88e9683640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88e40b3ee0 0x7f88e41f3450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f88d400c3f0 tx=0x7f88d400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88d9ffb640 1 -- 172.21.15.77:0/659800391 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d40074e0 con 0x7f88e40b3ee0 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88d9ffb640 1 -- 172.21.15.77:0/659800391 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d40047e0 con 0x7f88e40b3ee0 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88d9ffb640 1 -- 172.21.15.77:0/659800391 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4004bd0 con 0x7f88e40b3ee0 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.851+0000 7f88eb8e7640 1 -- 172.21.15.77:0/659800391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e415b430 con 0x7f88e40b3ee0 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.852+0000 7f88eb8e7640 1 -- 172.21.15.77:0/659800391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e40cb210 con 0x7f88e40b3ee0 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.853+0000 7f88eb8e7640 1 -- 172.21.15.77:0/659800391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e4068fb0 con 0x7f88e40b3ee0 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.853+0000 7f88d9ffb640 1 -- 172.21.15.77:0/659800391 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f88d4011070 con 0x7f88e40b3ee0 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.853+0000 7f88d9ffb640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f88c806be30 0x7f88c806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:38.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.853+0000 7f88d9ffb640 1 -- 172.21.15.77:0/659800391 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f88d408a7f0 con 0x7f88e40b3ee0 2024-04-25T01:21:38.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.853+0000 7f88e8e82640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f88c806be30 0x7f88c806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:38.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.854+0000 7f88e8e82640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f88c806be30 0x7f88c806e2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f88e0034000 tx=0x7f88e002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:38.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:38.857+0000 7f88d9ffb640 1 -- 172.21.15.77:0/659800391 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f88d40593e0 con 0x7f88e40b3ee0 2024-04-25T01:21:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.031+0000 7f88eb8e7640 1 -- 172.21.15.77:0/659800391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88e40cee20 con 0x7f88e40b3ee0 2024-04-25T01:21:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.032+0000 7f88d9ffb640 1 -- 172.21.15.77:0/659800391 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12884 (secure 0 0 0) 0x7f88d4058ce0 con 0x7f88e40b3ee0 2024-04-25T01:21:39.014 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:39.014 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":16,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:35.242898+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:21:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 -- 172.21.15.77:0/659800391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f88c806be30 msgr2=0x7f88c806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f88c806be30 0x7f88c806e2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f88e0034000 tx=0x7f88e002e000 comp rx=0 tx=0).stop 2024-04-25T01:21:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 -- 172.21.15.77:0/659800391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88e40b3ee0 msgr2=0x7f88e41f3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88e40b3ee0 0x7f88e41f3450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f88d400c3f0 tx=0x7f88d400c8b0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 -- 172.21.15.77:0/659800391 shutdown_connections 2024-04-25T01:21:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f88c806be30 0x7f88c806e2f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88e415e4c0 0x7f88e40b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88e40d4a10 0x7f88e40c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 --2- 172.21.15.77:0/659800391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88e40b3ee0 0x7f88e41f3450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 -- 172.21.15.77:0/659800391 >> 172.21.15.77:0/659800391 conn(0x7f88e40d3ef0 msgr2=0x7f88e415d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 -- 172.21.15.77:0/659800391 shutdown_connections 2024-04-25T01:21:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.034+0000 7f88eb8e7640 1 -- 172.21.15.77:0/659800391 wait complete. 2024-04-25T01:21:39.027 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:21:39.027 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:21:39.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.145+0000 7f5f8d8b2640 1 Processor -- start 2024-04-25T01:21:39.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.145+0000 7f5f8d8b2640 1 -- start start 2024-04-25T01:21:39.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f8d8b2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f880d23d0 0x7f5f880d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f8d8b2640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5f880d27b0 0x7f5f880d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f8d8b2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f880cea40 con 0x7f5f880b9380 2024-04-25T01:21:39.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f8d8b2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f5f880cece0 con 0x7f5f880d23d0 2024-04-25T01:21:39.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f8d8b2640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5f880cefc0 con 0x7f5f880d27b0 2024-04-25T01:21:39.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f86ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f880d23d0 0x7f5f880d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f86ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f880d23d0 0x7f5f880d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53010/0 (socket says 172.21.15.77:53010) 2024-04-25T01:21:39.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f86ffd640 1 -- 172.21.15.77:0/1108142279 learned_addr learned my addr 172.21.15.77:0/1108142279 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:39.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f867fc640 1 --2- 172.21.15.77:0/1108142279 >> v2:172.21.15.121:3300/0 conn(0x7f5f880d27b0 0x7f5f880d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f86ffd640 1 -- 172.21.15.77:0/1108142279 >> v2:172.21.15.121:3300/0 conn(0x7f5f880d27b0 msgr2=0x7f5f880d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:39.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f86ffd640 1 --2- 172.21.15.77:0/1108142279 >> v2:172.21.15.121:3300/0 conn(0x7f5f880d27b0 0x7f5f880d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f86ffd640 1 -- 172.21.15.77:0/1108142279 >> v1:172.21.15.121:6789/0 conn(0x7f5f880b9380 legacy=0x7f5f880b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:39.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.146+0000 7f5f86ffd640 1 -- 172.21.15.77:0/1108142279 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f880cf2a0 con 0x7f5f880d23d0 2024-04-25T01:21:39.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.147+0000 7f5f86ffd640 1 --2- 172.21.15.77:0/1108142279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f880d23d0 0x7f5f880d4920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f88067db0 tx=0x7f5f7802a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e3b57fc1b6c976 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:39.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.147+0000 7f5f85ffb640 1 -- 172.21.15.77:0/1108142279 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78036070 con 0x7f5f880d23d0 2024-04-25T01:21:39.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.147+0000 7f5f85ffb640 1 -- 172.21.15.77:0/1108142279 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f78031600 con 0x7f5f880d23d0 2024-04-25T01:21:39.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.147+0000 7f5f85ffb640 1 -- 172.21.15.77:0/1108142279 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f780319d0 con 0x7f5f880d23d0 2024-04-25T01:21:39.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.147+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1108142279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f880d23d0 msgr2=0x7f5f880d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.147+0000 7f5f8d8b2640 1 --2- 172.21.15.77:0/1108142279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f880d23d0 0x7f5f880d4920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f88067db0 tx=0x7f5f7802a750 comp rx=0 tx=0).stop 2024-04-25T01:21:39.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.148+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1108142279 shutdown_connections 2024-04-25T01:21:39.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.148+0000 7f5f8d8b2640 1 --2- 172.21.15.77:0/1108142279 >> v2:172.21.15.121:3300/0 conn(0x7f5f880d27b0 0x7f5f880d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.148+0000 7f5f8d8b2640 1 --2- 172.21.15.77:0/1108142279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f880d23d0 0x7f5f880d4920 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.148+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1108142279 >> 172.21.15.77:0/1108142279 conn(0x7f5f880d3e30 msgr2=0x7f5f880d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:39.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.148+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1108142279 shutdown_connections 2024-04-25T01:21:39.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.148+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1108142279 wait complete. 2024-04-25T01:21:39.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.149+0000 7f5f8d8b2640 1 Processor -- start 2024-04-25T01:21:39.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.149+0000 7f5f8d8b2640 1 -- start start 2024-04-25T01:21:39.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f8d8b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f880b9380 0x7f5f8815b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f8d8b2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f880d23d0 0x7f5f8806ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f8d8b2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f880d27b0 0x7f5f8806cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f8d8b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5f880cfd80 con 0x7f5f880b9380 2024-04-25T01:21:39.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f8d8b2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5f880cf860 con 0x7f5f880d23d0 2024-04-25T01:21:39.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f8d8b2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5f880b6c80 con 0x7f5f880d27b0 2024-04-25T01:21:39.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f86ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f880b9380 0x7f5f8815b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f86ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f880b9380 0x7f5f8815b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53012/0 (socket says 172.21.15.77:53012) 2024-04-25T01:21:39.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f86ffd640 1 -- 172.21.15.77:0/1820100050 learned_addr learned my addr 172.21.15.77:0/1820100050 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f867fc640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f880d23d0 0x7f5f8806ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f877fe640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f880d27b0 0x7f5f8806cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f86ffd640 1 -- 172.21.15.77:0/1820100050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f880d27b0 msgr2=0x7f5f8806cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f86ffd640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f880d27b0 0x7f5f8806cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f86ffd640 1 -- 172.21.15.77:0/1820100050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f880d23d0 msgr2=0x7f5f8806ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f86ffd640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f880d23d0 0x7f5f8806ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f86ffd640 1 -- 172.21.15.77:0/1820100050 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f8815cbd0 con 0x7f5f880b9380 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f86ffd640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f880b9380 0x7f5f8815b2e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009f10 tx=0x7f5f78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f6ffff640 1 -- 172.21.15.77:0/1820100050 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78036050 con 0x7f5f880b9380 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f6ffff640 1 -- 172.21.15.77:0/1820100050 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f78042070 con 0x7f5f880b9380 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f6ffff640 1 -- 172.21.15.77:0/1820100050 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78009510 con 0x7f5f880b9380 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1820100050 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f8815ceb0 con 0x7f5f880b9380 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.150+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1820100050 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f8815d470 con 0x7f5f880b9380 2024-04-25T01:21:39.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.151+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1820100050 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f88068fb0 con 0x7f5f880b9380 2024-04-25T01:21:39.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.152+0000 7f5f6ffff640 1 -- 172.21.15.77:0/1820100050 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5f78007c50 con 0x7f5f880b9380 2024-04-25T01:21:39.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.152+0000 7f5f6ffff640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f7406be60 0x7f5f7406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.152+0000 7f5f867fc640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f7406be60 0x7f5f7406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.152+0000 7f5f6ffff640 1 -- 172.21.15.77:0/1820100050 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f5f780ab8c0 con 0x7f5f880b9380 2024-04-25T01:21:39.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.153+0000 7f5f867fc640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f7406be60 0x7f5f7406e320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5f880b4230 tx=0x7f5f7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:39.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.156+0000 7f5f6ffff640 1 -- 172.21.15.77:0/1820100050 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5f7807a120 con 0x7f5f880b9380 2024-04-25T01:21:39.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.331+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1820100050 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5f88161680 con 0x7f5f880b9380 2024-04-25T01:21:39.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.331+0000 7f5f6ffff640 1 -- 172.21.15.77:0/1820100050 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f5f7807e420 con 0x7f5f880b9380 2024-04-25T01:21:39.313 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:21:39.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1820100050 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f7406be60 msgr2=0x7f5f7406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f7406be60 0x7f5f7406e320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5f880b4230 tx=0x7f5f7c008040 comp rx=0 tx=0).stop 2024-04-25T01:21:39.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1820100050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f880b9380 msgr2=0x7f5f8815b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f880b9380 0x7f5f8815b2e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009f10 tx=0x7f5f78004a40 comp rx=0 tx=0).stop 2024-04-25T01:21:39.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1820100050 shutdown_connections 2024-04-25T01:21:39.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f7406be60 0x7f5f7406e320 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f880d27b0 0x7f5f8806cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f880d23d0 0x7f5f8806ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 --2- 172.21.15.77:0/1820100050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f880b9380 0x7f5f8815b2e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1820100050 >> 172.21.15.77:0/1820100050 conn(0x7f5f880d3e30 msgr2=0x7f5f880ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:39.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1820100050 shutdown_connections 2024-04-25T01:21:39.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.333+0000 7f5f8d8b2640 1 -- 172.21.15.77:0/1820100050 wait complete. 2024-04-25T01:21:39.325 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:21:39.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.439+0000 7f62d845d640 1 Processor -- start 2024-04-25T01:21:39.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.439+0000 7f62d845d640 1 -- start start 2024-04-25T01:21:39.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.440+0000 7f62d845d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f62d015e380 0x7f62d00cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.440+0000 7f62d845d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d00d1070 0x7f62d00b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.440+0000 7f62d845d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62d00d1eb0 con 0x7f62d00cfcf0 2024-04-25T01:21:39.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.440+0000 7f62d845d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f62d00d2190 con 0x7f62d015e380 2024-04-25T01:21:39.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.440+0000 7f62d845d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62d00d2470 con 0x7f62d00d1070 2024-04-25T01:21:39.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.440+0000 7f62d61f9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f62d015e380 0x7f62d00cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.440+0000 7f62d59f8640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f62d00cfcf0 0x7f62d00d0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:40346/0 (socket says 172.21.15.77:40346) 2024-04-25T01:21:39.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.440+0000 7f62d59f8640 1 -- 172.21.15.77:0/3791022610 learned_addr learned my addr 172.21.15.77:0/3791022610 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:39.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.441+0000 7f62d69fa640 1 --2- 172.21.15.77:0/3791022610 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d00d1070 0x7f62d00b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.441+0000 7f62d61f9640 1 -- 172.21.15.77:0/3791022610 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d00d1070 msgr2=0x7f62d00b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:39.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.441+0000 7f62d61f9640 1 --2- 172.21.15.77:0/3791022610 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d00d1070 0x7f62d00b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.441+0000 7f62d61f9640 1 -- 172.21.15.77:0/3791022610 >> v1:172.21.15.77:6789/0 conn(0x7f62d00cfcf0 legacy=0x7f62d00d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.441+0000 7f62d61f9640 1 -- 172.21.15.77:0/3791022610 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d00d2750 con 0x7f62d015e380 2024-04-25T01:21:39.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.441+0000 7f62d51f7640 1 -- 172.21.15.77:0/3791022610 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 76761271 0 0) 0x7f62d00d1eb0 con 0x7f62d00cfcf0 2024-04-25T01:21:39.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.441+0000 7f62d61f9640 1 --2- 172.21.15.77:0/3791022610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d015e380 0x7f62d00cf7b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009d90 tx=0x7f62cc02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17cb63e9f4ebf9b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:39.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.442+0000 7f62d51f7640 1 -- 172.21.15.77:0/3791022610 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc007740 con 0x7f62d015e380 2024-04-25T01:21:39.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.442+0000 7f62d51f7640 1 -- 172.21.15.77:0/3791022610 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62cc0078e0 con 0x7f62d015e380 2024-04-25T01:21:39.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.442+0000 7f62d51f7640 1 -- 172.21.15.77:0/3791022610 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc007cb0 con 0x7f62d015e380 2024-04-25T01:21:39.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.442+0000 7f62d845d640 1 -- 172.21.15.77:0/3791022610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d015e380 msgr2=0x7f62d00cf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.442+0000 7f62d845d640 1 --2- 172.21.15.77:0/3791022610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d015e380 0x7f62d00cf7b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009d90 tx=0x7f62cc02a660 comp rx=0 tx=0).stop 2024-04-25T01:21:39.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.442+0000 7f62d845d640 1 -- 172.21.15.77:0/3791022610 shutdown_connections 2024-04-25T01:21:39.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.442+0000 7f62d845d640 1 --2- 172.21.15.77:0/3791022610 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d00d1070 0x7f62d00b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.442+0000 7f62d845d640 1 --2- 172.21.15.77:0/3791022610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d015e380 0x7f62d00cf7b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.442+0000 7f62d845d640 1 -- 172.21.15.77:0/3791022610 >> 172.21.15.77:0/3791022610 conn(0x7f62d009d200 msgr2=0x7f62d00d3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:39.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.442+0000 7f62d845d640 1 -- 172.21.15.77:0/3791022610 shutdown_connections 2024-04-25T01:21:39.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.442+0000 7f62d845d640 1 -- 172.21.15.77:0/3791022610 wait complete. 2024-04-25T01:21:39.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.443+0000 7f62d845d640 1 Processor -- start 2024-04-25T01:21:39.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.443+0000 7f62d845d640 1 -- start start 2024-04-25T01:21:39.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d845d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d00cfcf0 0x7f62d015cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d845d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d00d1070 0x7f62d00b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d845d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d015e380 0x7f62d015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d845d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f62d00d2f50 con 0x7f62d015e380 2024-04-25T01:21:39.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d845d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f62d00d1b40 con 0x7f62d00cfcf0 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d845d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62d00d30d0 con 0x7f62d00d1070 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d69fa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d015e380 0x7f62d015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d69fa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d015e380 0x7f62d015c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53042/0 (socket says 172.21.15.77:53042) 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d69fa640 1 -- 172.21.15.77:0/3029862838 learned_addr learned my addr 172.21.15.77:0/3029862838 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d59f8640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d00d1070 0x7f62d00b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d61f9640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d00cfcf0 0x7f62d015cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d69fa640 1 -- 172.21.15.77:0/3029862838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d00d1070 msgr2=0x7f62d00b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d69fa640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d00d1070 0x7f62d00b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d69fa640 1 -- 172.21.15.77:0/3029862838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d00cfcf0 msgr2=0x7f62d015cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d69fa640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d00cfcf0 0x7f62d015cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.444+0000 7f62d69fa640 1 -- 172.21.15.77:0/3029862838 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d00cc5e0 con 0x7f62d015e380 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.445+0000 7f62d69fa640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d015e380 0x7f62d015c2b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f62c800d830 tx=0x7f62c800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.445+0000 7f62c6ffd640 1 -- 172.21.15.77:0/3029862838 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c800b6c0 con 0x7f62d015e380 2024-04-25T01:21:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.445+0000 7f62c6ffd640 1 -- 172.21.15.77:0/3029862838 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62c8019070 con 0x7f62d015e380 2024-04-25T01:21:39.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.445+0000 7f62d845d640 1 -- 172.21.15.77:0/3029862838 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d01f7ef0 con 0x7f62d015e380 2024-04-25T01:21:39.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.445+0000 7f62c6ffd640 1 -- 172.21.15.77:0/3029862838 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c800bac0 con 0x7f62d015e380 2024-04-25T01:21:39.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.445+0000 7f62d845d640 1 -- 172.21.15.77:0/3029862838 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d01f84b0 con 0x7f62d015e380 2024-04-25T01:21:39.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.446+0000 7f62d845d640 1 -- 172.21.15.77:0/3029862838 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d01f7c70 con 0x7f62d015e380 2024-04-25T01:21:39.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.446+0000 7f62c6ffd640 1 -- 172.21.15.77:0/3029862838 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f62c801e460 con 0x7f62d015e380 2024-04-25T01:21:39.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.446+0000 7f62c6ffd640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62b406be30 0x7f62b406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:39.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.447+0000 7f62c6ffd640 1 -- 172.21.15.77:0/3029862838 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f62c808b1b0 con 0x7f62d015e380 2024-04-25T01:21:39.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.447+0000 7f62d61f9640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62b406be30 0x7f62b406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:39.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.447+0000 7f62d61f9640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62b406be30 0x7f62b406e2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f62cc034000 tx=0x7f62cc033040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:39.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.451+0000 7f62c6ffd640 1 -- 172.21.15.77:0/3029862838 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f62d01f7c70 con 0x7f62d015e380 2024-04-25T01:21:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.659+0000 7f62d845d640 1 -- 172.21.15.77:0/3029862838 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62d0068180 con 0x7f62d015e380 2024-04-25T01:21:39.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.660+0000 7f62c6ffd640 1 -- 172.21.15.77:0/3029862838 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f62c8059eb0 con 0x7f62d015e380 2024-04-25T01:21:39.642 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:39.642 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:21:39.642 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 2 2024-04-25T01:21:39.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 -- 172.21.15.77:0/3029862838 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62b406be30 msgr2=0x7f62b406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62b406be30 0x7f62b406e2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f62cc034000 tx=0x7f62cc033040 comp rx=0 tx=0).stop 2024-04-25T01:21:39.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 -- 172.21.15.77:0/3029862838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d015e380 msgr2=0x7f62d015c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:39.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d015e380 0x7f62d015c2b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f62c800d830 tx=0x7f62c800dcf0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 -- 172.21.15.77:0/3029862838 shutdown_connections 2024-04-25T01:21:39.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62b406be30 0x7f62b406e2f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d015e380 0x7f62d015c2b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d00d1070 0x7f62d00b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 --2- 172.21.15.77:0/3029862838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d00cfcf0 0x7f62d015cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:39.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 -- 172.21.15.77:0/3029862838 >> 172.21.15.77:0/3029862838 conn(0x7f62d009d200 msgr2=0x7f62d006c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:39.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 -- 172.21.15.77:0/3029862838 shutdown_connections 2024-04-25T01:21:39.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:39.662+0000 7f62d845d640 1 -- 172.21.15.77:0/3029862838 wait complete. 2024-04-25T01:21:41.655 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:21:41.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.780+0000 7f9e14563640 1 Processor -- start 2024-04-25T01:21:41.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.780+0000 7f9e14563640 1 -- start start 2024-04-25T01:21:41.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.781+0000 7f9e14563640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9e0c0cff30 0x7f9e0c0d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:41.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.781+0000 7f9e14563640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e0c0d0850 con 0x7f9e0c15e420 2024-04-25T01:21:41.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.781+0000 7f9e14563640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e0c06d740 con 0x7f9e0c06d240 2024-04-25T01:21:41.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.781+0000 7f9e14563640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f9e0c06da20 con 0x7f9e0c0cff30 2024-04-25T01:21:41.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.781+0000 7f9e122ff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9e0c0cff30 0x7f9e0c0d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:41.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.781+0000 7f9e11afe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9e0c15e420 0x7f9e0c0d0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:40358/0 (socket says 172.21.15.77:40358) 2024-04-25T01:21:41.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.781+0000 7f9e11afe640 1 -- 172.21.15.77:0/3906527571 learned_addr learned my addr 172.21.15.77:0/3906527571 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:41.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.782+0000 7f9e122ff640 1 -- 172.21.15.77:0/3906527571 >> v1:172.21.15.121:6789/0 conn(0x7f9e0c06d240 legacy=0x7f9e0c0aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:41.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.782+0000 7f9e122ff640 1 -- 172.21.15.77:0/3906527571 >> v1:172.21.15.77:6789/0 conn(0x7f9e0c15e420 legacy=0x7f9e0c0d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:41.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.782+0000 7f9e122ff640 1 -- 172.21.15.77:0/3906527571 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0c06dd00 con 0x7f9e0c0cff30 2024-04-25T01:21:41.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.782+0000 7f9e112fd640 1 -- 172.21.15.77:0/3906527571 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3984471337 0 0) 0x7f9e0c0d0850 con 0x7f9e0c15e420 2024-04-25T01:21:41.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.782+0000 7f9e122ff640 1 --2- 172.21.15.77:0/3906527571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0cff30 0x7f9e0c0d0310 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9e080097a0 tx=0x7f9e0802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c461bc66cb874c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:41.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.782+0000 7f9e112fd640 1 -- 172.21.15.77:0/3906527571 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08037070 con 0x7f9e0c0cff30 2024-04-25T01:21:41.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.782+0000 7f9e112fd640 1 -- 172.21.15.77:0/3906527571 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e08032440 con 0x7f9e0c0cff30 2024-04-25T01:21:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.783+0000 7f9e112fd640 1 -- 172.21.15.77:0/3906527571 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08032810 con 0x7f9e0c0cff30 2024-04-25T01:21:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.783+0000 7f9e14563640 1 -- 172.21.15.77:0/3906527571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0cff30 msgr2=0x7f9e0c0d0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.783+0000 7f9e14563640 1 --2- 172.21.15.77:0/3906527571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0cff30 0x7f9e0c0d0310 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9e080097a0 tx=0x7f9e0802a1e0 comp rx=0 tx=0).stop 2024-04-25T01:21:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.783+0000 7f9e14563640 1 -- 172.21.15.77:0/3906527571 shutdown_connections 2024-04-25T01:21:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.783+0000 7f9e14563640 1 --2- 172.21.15.77:0/3906527571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0cff30 0x7f9e0c0d0310 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.783+0000 7f9e14563640 1 -- 172.21.15.77:0/3906527571 >> 172.21.15.77:0/3906527571 conn(0x7f9e0c0d1ca0 msgr2=0x7f9e0c0d20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.783+0000 7f9e14563640 1 -- 172.21.15.77:0/3906527571 shutdown_connections 2024-04-25T01:21:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.783+0000 7f9e14563640 1 -- 172.21.15.77:0/3906527571 wait complete. 2024-04-25T01:21:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.784+0000 7f9e14563640 1 Processor -- start 2024-04-25T01:21:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.784+0000 7f9e14563640 1 -- start start 2024-04-25T01:21:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.785+0000 7f9e14563640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c06d240 0x7f9e0c15b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.785+0000 7f9e14563640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e0c0cff30 0x7f9e0c1f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.785+0000 7f9e14563640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e0c15e420 0x7f9e0c1f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.785+0000 7f9e14563640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0c0af660 con 0x7f9e0c06d240 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.785+0000 7f9e14563640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0c0af4e0 con 0x7f9e0c15e420 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.785+0000 7f9e14563640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9e0c0af7e0 con 0x7f9e0c0cff30 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.785+0000 7f9e122ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c06d240 0x7f9e0c15b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.785+0000 7f9e122ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c06d240 0x7f9e0c15b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53060/0 (socket says 172.21.15.77:53060) 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.785+0000 7f9e122ff640 1 -- 172.21.15.77:0/1845870409 learned_addr learned my addr 172.21.15.77:0/1845870409 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.785+0000 7f9e12b00640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e0c15e420 0x7f9e0c1f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.785+0000 7f9e11afe640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e0c0cff30 0x7f9e0c1f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e122ff640 1 -- 172.21.15.77:0/1845870409 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e0c0cff30 msgr2=0x7f9e0c1f4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e122ff640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e0c0cff30 0x7f9e0c1f4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e122ff640 1 -- 172.21.15.77:0/1845870409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e0c15e420 msgr2=0x7f9e0c1f4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e122ff640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e0c15e420 0x7f9e0c1f4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e122ff640 1 -- 172.21.15.77:0/1845870409 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0c158b80 con 0x7f9e0c06d240 2024-04-25T01:21:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e12b00640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e0c15e420 0x7f9e0c1f4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e122ff640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c06d240 0x7f9e0c15b4f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9e080098c0 tx=0x7f9e080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e02ffd640 1 -- 172.21.15.77:0/1845870409 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08041070 con 0x7f9e0c06d240 2024-04-25T01:21:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e02ffd640 1 -- 172.21.15.77:0/1845870409 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0803c870 con 0x7f9e0c06d240 2024-04-25T01:21:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e14563640 1 -- 172.21.15.77:0/1845870409 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e0c158e60 con 0x7f9e0c06d240 2024-04-25T01:21:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e02ffd640 1 -- 172.21.15.77:0/1845870409 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08037040 con 0x7f9e0c06d240 2024-04-25T01:21:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.786+0000 7f9e14563640 1 -- 172.21.15.77:0/1845870409 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e0c159420 con 0x7f9e0c06d240 2024-04-25T01:21:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.787+0000 7f9e14563640 1 -- 172.21.15.77:0/1845870409 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e0c068fb0 con 0x7f9e0c06d240 2024-04-25T01:21:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.787+0000 7f9e02ffd640 1 -- 172.21.15.77:0/1845870409 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9e08044070 con 0x7f9e0c06d240 2024-04-25T01:21:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.788+0000 7f9e02ffd640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9df006be30 0x7f9df006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.788+0000 7f9e11afe640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9df006be30 0x7f9df006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.788+0000 7f9e02ffd640 1 -- 172.21.15.77:0/1845870409 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f9e080ab650 con 0x7f9e0c06d240 2024-04-25T01:21:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.788+0000 7f9e11afe640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9df006be30 0x7f9df006e2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0b7d20 tx=0x7f9dfc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:41.792+0000 7f9e02ffd640 1 -- 172.21.15.77:0/1845870409 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9e0807a470 con 0x7f9e0c06d240 2024-04-25T01:21:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.001+0000 7f9e14563640 1 -- 172.21.15.77:0/1845870409 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e0c0b1600 con 0x7f9e0c06d240 2024-04-25T01:21:41.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.002+0000 7f9e02ffd640 1 -- 172.21.15.77:0/1845870409 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3867 (secure 0 0 0) 0x7f9e08079d70 con 0x7f9e0c06d240 2024-04-25T01:21:41.984 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:41.984 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4267907621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4267907621},{"type":"v1","addr":"172.21.15.77:6837","nonce":4267907621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/493518847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":493518847},{"type":"v1","addr":"172.21.15.121:6833","nonce":493518847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2810316639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2810316639},{"type":"v1","addr":"172.21.15.121:6835","nonce":2810316639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T01:21:41.984 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 3 2024-04-25T01:21:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 -- 172.21.15.77:0/1845870409 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9df006be30 msgr2=0x7f9df006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9df006be30 0x7f9df006e2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0b7d20 tx=0x7f9dfc009040 comp rx=0 tx=0).stop 2024-04-25T01:21:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 -- 172.21.15.77:0/1845870409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c06d240 msgr2=0x7f9e0c15b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c06d240 0x7f9e0c15b4f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9e080098c0 tx=0x7f9e080046c0 comp rx=0 tx=0).stop 2024-04-25T01:21:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 -- 172.21.15.77:0/1845870409 shutdown_connections 2024-04-25T01:21:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9df006be30 0x7f9df006e2f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e0c15e420 0x7f9e0c1f4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e0c0cff30 0x7f9e0c1f4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 --2- 172.21.15.77:0/1845870409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c06d240 0x7f9e0c15b4f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 -- 172.21.15.77:0/1845870409 >> 172.21.15.77:0/1845870409 conn(0x7f9e0c0d1ca0 msgr2=0x7f9e0c0b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 -- 172.21.15.77:0/1845870409 shutdown_connections 2024-04-25T01:21:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.004+0000 7f9e14563640 1 -- 172.21.15.77:0/1845870409 wait complete. 2024-04-25T01:21:41.996 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:21:42.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.111+0000 7fbee7b63640 1 Processor -- start 2024-04-25T01:21:42.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.111+0000 7fbee7b63640 1 -- start start 2024-04-25T01:21:42.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee7b63640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee006fcd0 0x7fbee015ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee7b63640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbee00ade10 0x7fbee006c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee7b63640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbee006cac0 con 0x7fbee006c670 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee7b63640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fbee01589e0 con 0x7fbee00ade10 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee7b63640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbee00b3890 con 0x7fbee006fcd0 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee50fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbee00ade10 0x7fbee006c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee50fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbee00ade10 0x7fbee006c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53072/0 (socket says 172.21.15.77:53072) 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee50fe640 1 -- 172.21.15.77:0/1481251995 learned_addr learned my addr 172.21.15.77:0/1481251995 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee58ff640 1 --2- 172.21.15.77:0/1481251995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee006fcd0 0x7fbee015ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee50fe640 1 -- 172.21.15.77:0/1481251995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee006fcd0 msgr2=0x7fbee015ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee50fe640 1 --2- 172.21.15.77:0/1481251995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee006fcd0 0x7fbee015ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee50fe640 1 -- 172.21.15.77:0/1481251995 >> v1:172.21.15.121:6789/0 conn(0x7fbee006c670 legacy=0x7fbee0161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee50fe640 1 -- 172.21.15.77:0/1481251995 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee00ae290 con 0x7fbee00ade10 2024-04-25T01:21:42.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.112+0000 7fbee58ff640 1 --2- 172.21.15.77:0/1481251995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee006fcd0 0x7fbee015ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee50fe640 1 --2- 172.21.15.77:0/1481251995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee00ade10 0x7fbee006c130 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009870 tx=0x7fbed002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe86483649d1d5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee48fd640 1 -- 172.21.15.77:0/1481251995 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0037070 con 0x7fbee00ade10 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee48fd640 1 -- 172.21.15.77:0/1481251995 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed0032440 con 0x7fbee00ade10 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee48fd640 1 -- 172.21.15.77:0/1481251995 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0032810 con 0x7fbee00ade10 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee7b63640 1 -- 172.21.15.77:0/1481251995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee00ade10 msgr2=0x7fbee006c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee7b63640 1 --2- 172.21.15.77:0/1481251995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee00ade10 0x7fbee006c130 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009870 tx=0x7fbed002a180 comp rx=0 tx=0).stop 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee7b63640 1 -- 172.21.15.77:0/1481251995 shutdown_connections 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee7b63640 1 --2- 172.21.15.77:0/1481251995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee00ade10 0x7fbee006c130 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee7b63640 1 --2- 172.21.15.77:0/1481251995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee006fcd0 0x7fbee015ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee7b63640 1 -- 172.21.15.77:0/1481251995 >> 172.21.15.77:0/1481251995 conn(0x7fbee0071550 msgr2=0x7fbee0071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee7b63640 1 -- 172.21.15.77:0/1481251995 shutdown_connections 2024-04-25T01:21:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.113+0000 7fbee7b63640 1 -- 172.21.15.77:0/1481251995 wait complete. 2024-04-25T01:21:42.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.115+0000 7fbee7b63640 1 Processor -- start 2024-04-25T01:21:42.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.115+0000 7fbee7b63640 1 -- start start 2024-04-25T01:21:42.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.115+0000 7fbee7b63640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee006c670 0x7fbee00b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.115+0000 7fbee7b63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee006fcd0 0x7fbee00ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.115+0000 7fbee7b63640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee00ade10 0x7fbee00cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.115+0000 7fbee7b63640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbee00ae7f0 con 0x7fbee006fcd0 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.115+0000 7fbee7b63640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbee00b3560 con 0x7fbee00ade10 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.115+0000 7fbee7b63640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbee00aead0 con 0x7fbee006c670 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.115+0000 7fbee50fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee006fcd0 0x7fbee00ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.115+0000 7fbee50fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee006fcd0 0x7fbee00ceb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53080/0 (socket says 172.21.15.77:53080) 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.115+0000 7fbee50fe640 1 -- 172.21.15.77:0/1967197425 learned_addr learned my addr 172.21.15.77:0/1967197425 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbee58ff640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee006c670 0x7fbee00b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbee6100640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee00ade10 0x7fbee00cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbee50fe640 1 -- 172.21.15.77:0/1967197425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee006c670 msgr2=0x7fbee00b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbee50fe640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee006c670 0x7fbee00b1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbee50fe640 1 -- 172.21.15.77:0/1967197425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee00ade10 msgr2=0x7fbee00cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbee50fe640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee00ade10 0x7fbee00cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbee50fe640 1 -- 172.21.15.77:0/1967197425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee01f7870 con 0x7fbee006fcd0 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbee50fe640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee006fcd0 0x7fbee00ceb90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbed00090b0 tx=0x7fbed00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbed67fc640 1 -- 172.21.15.77:0/1967197425 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0041020 con 0x7fbee006fcd0 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbed67fc640 1 -- 172.21.15.77:0/1967197425 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed0030990 con 0x7fbee006fcd0 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbee7b63640 1 -- 172.21.15.77:0/1967197425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee01f7b50 con 0x7fbee006fcd0 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbed67fc640 1 -- 172.21.15.77:0/1967197425 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0037040 con 0x7fbee006fcd0 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.116+0000 7fbee7b63640 1 -- 172.21.15.77:0/1967197425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee01f8110 con 0x7fbee006fcd0 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.117+0000 7fbee7b63640 1 -- 172.21.15.77:0/1967197425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee0068fb0 con 0x7fbee006fcd0 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.117+0000 7fbed67fc640 1 -- 172.21.15.77:0/1967197425 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbed0044020 con 0x7fbee006fcd0 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.118+0000 7fbed67fc640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec406be30 0x7fbec406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.118+0000 7fbed67fc640 1 -- 172.21.15.77:0/1967197425 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fbed00ab510 con 0x7fbee006fcd0 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.118+0000 7fbee58ff640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec406be30 0x7fbec406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.118+0000 7fbee58ff640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec406be30 0x7fbec406e2f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbedc008850 tx=0x7fbedc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.122+0000 7fbed67fc640 1 -- 172.21.15.77:0/1967197425 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbed004b030 con 0x7fbee006fcd0 2024-04-25T01:21:42.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.296+0000 7fbee7b63640 1 -- 172.21.15.77:0/1967197425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbee00b52c0 con 0x7fbee006fcd0 2024-04-25T01:21:42.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.296+0000 7fbed67fc640 1 -- 172.21.15.77:0/1967197425 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12884 (secure 0 0 0) 0x7fbed00ab070 con 0x7fbee006fcd0 2024-04-25T01:21:42.279 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:42.279 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":16,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:35.242898+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.298+0000 7fbee7b63640 1 -- 172.21.15.77:0/1967197425 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec406be30 msgr2=0x7fbec406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.298+0000 7fbee7b63640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec406be30 0x7fbec406e2f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbedc008850 tx=0x7fbedc008040 comp rx=0 tx=0).stop 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.298+0000 7fbee7b63640 1 -- 172.21.15.77:0/1967197425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee006fcd0 msgr2=0x7fbee00ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.298+0000 7fbee7b63640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee006fcd0 0x7fbee00ceb90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbed00090b0 tx=0x7fbed00046c0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.298+0000 7fbee7b63640 1 -- 172.21.15.77:0/1967197425 shutdown_connections 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.298+0000 7fbee7b63640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec406be30 0x7fbec406e2f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.298+0000 7fbee7b63640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee006fcd0 0x7fbee00ceb90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.298+0000 7fbee7b63640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee00ade10 0x7fbee00cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.298+0000 7fbee7b63640 1 --2- 172.21.15.77:0/1967197425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee006c670 0x7fbee00b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.298+0000 7fbee7b63640 1 -- 172.21.15.77:0/1967197425 >> 172.21.15.77:0/1967197425 conn(0x7fbee0071550 msgr2=0x7fbee00b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.299+0000 7fbee7b63640 1 -- 172.21.15.77:0/1967197425 shutdown_connections 2024-04-25T01:21:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.299+0000 7fbee7b63640 1 -- 172.21.15.77:0/1967197425 wait complete. 2024-04-25T01:21:42.291 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:21:42.291 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:21:42.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.409+0000 7f21a9309640 1 Processor -- start 2024-04-25T01:21:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.409+0000 7f21a9309640 1 -- start start 2024-04-25T01:21:42.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a9309640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f21a40b4ac0 0x7f21a4160300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a9309640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f21a40b2be0 0x7f21a40cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a9309640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21a4160840 con 0x7f21a415f6a0 2024-04-25T01:21:42.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a9309640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f21a4160b00 con 0x7f21a40b4ac0 2024-04-25T01:21:42.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a9309640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f21a40cf840 con 0x7f21a40b2be0 2024-04-25T01:21:42.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a27fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f21a40b4ac0 0x7f21a4160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a27fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f21a40b4ac0 0x7f21a4160300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53094/0 (socket says 172.21.15.77:53094) 2024-04-25T01:21:42.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a27fc640 1 -- 172.21.15.77:0/140168613 learned_addr learned my addr 172.21.15.77:0/140168613 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:42.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a27fc640 1 -- 172.21.15.77:0/140168613 >> v2:172.21.15.121:3300/0 conn(0x7f21a40b2be0 msgr2=0x7f21a40cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a27fc640 1 --2- 172.21.15.77:0/140168613 >> v2:172.21.15.121:3300/0 conn(0x7f21a40b2be0 0x7f21a40cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a27fc640 1 -- 172.21.15.77:0/140168613 >> v1:172.21.15.121:6789/0 conn(0x7f21a415f6a0 legacy=0x7f21a415fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.410+0000 7f21a27fc640 1 -- 172.21.15.77:0/140168613 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a40cd380 con 0x7f21a40b4ac0 2024-04-25T01:21:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.411+0000 7f21a27fc640 1 --2- 172.21.15.77:0/140168613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21a40b4ac0 0x7f21a4160300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f21a4071af0 tx=0x7f219802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1d2d3e1a999033 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.411+0000 7f21a1ffb640 1 -- 172.21.15.77:0/140168613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198037070 con 0x7f21a40b4ac0 2024-04-25T01:21:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.411+0000 7f21a1ffb640 1 -- 172.21.15.77:0/140168613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f219802ae20 con 0x7f21a40b4ac0 2024-04-25T01:21:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.411+0000 7f21a1ffb640 1 -- 172.21.15.77:0/140168613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198032660 con 0x7f21a40b4ac0 2024-04-25T01:21:42.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.412+0000 7f21a9309640 1 -- 172.21.15.77:0/140168613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21a40b4ac0 msgr2=0x7f21a4160300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.412+0000 7f21a9309640 1 --2- 172.21.15.77:0/140168613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21a40b4ac0 0x7f21a4160300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f21a4071af0 tx=0x7f219802a110 comp rx=0 tx=0).stop 2024-04-25T01:21:42.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.412+0000 7f21a9309640 1 -- 172.21.15.77:0/140168613 shutdown_connections 2024-04-25T01:21:42.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.412+0000 7f21a9309640 1 --2- 172.21.15.77:0/140168613 >> v2:172.21.15.121:3300/0 conn(0x7f21a40b2be0 0x7f21a40cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.412+0000 7f21a9309640 1 --2- 172.21.15.77:0/140168613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21a40b4ac0 0x7f21a4160300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.412+0000 7f21a9309640 1 -- 172.21.15.77:0/140168613 >> 172.21.15.77:0/140168613 conn(0x7f21a40b37a0 msgr2=0x7f21a40b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:42.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.412+0000 7f21a9309640 1 -- 172.21.15.77:0/140168613 shutdown_connections 2024-04-25T01:21:42.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.412+0000 7f21a9309640 1 -- 172.21.15.77:0/140168613 wait complete. 2024-04-25T01:21:42.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.413+0000 7f21a9309640 1 Processor -- start 2024-04-25T01:21:42.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.413+0000 7f21a9309640 1 -- start start 2024-04-25T01:21:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a9309640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21a40b2be0 0x7f21a4156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a9309640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21a40b4ac0 0x7f21a41e7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a9309640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21a415f6a0 0x7f21a41e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a9309640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f21a40b42c0 con 0x7f21a40b4ac0 2024-04-25T01:21:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a9309640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f21a40cd940 con 0x7f21a415f6a0 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a9309640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f21a40b4440 con 0x7f21a40b2be0 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21a40b4ac0 0x7f21a41e7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21a40b4ac0 0x7f21a41e7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53108/0 (socket says 172.21.15.77:53108) 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a27fc640 1 -- 172.21.15.77:0/2645464930 learned_addr learned my addr 172.21.15.77:0/2645464930 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a27fc640 1 -- 172.21.15.77:0/2645464930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21a40b2be0 msgr2=0x7f21a4156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a2ffd640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21a40b2be0 0x7f21a4156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a37fe640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21a415f6a0 0x7f21a41e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a27fc640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21a40b2be0 0x7f21a4156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a27fc640 1 -- 172.21.15.77:0/2645464930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21a415f6a0 msgr2=0x7f21a41e78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a27fc640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21a415f6a0 0x7f21a41e78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a27fc640 1 -- 172.21.15.77:0/2645464930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a41577c0 con 0x7f21a40b4ac0 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a27fc640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21a40b4ac0 0x7f21a41e7390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2198000c00 tx=0x7f21980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f218b7fe640 1 -- 172.21.15.77:0/2645464930 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198042070 con 0x7f21a40b4ac0 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f218b7fe640 1 -- 172.21.15.77:0/2645464930 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f219803e670 con 0x7f21a40b4ac0 2024-04-25T01:21:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f218b7fe640 1 -- 172.21.15.77:0/2645464930 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198037040 con 0x7f21a40b4ac0 2024-04-25T01:21:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.414+0000 7f21a9309640 1 -- 172.21.15.77:0/2645464930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a40c9920 con 0x7f21a40b4ac0 2024-04-25T01:21:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.415+0000 7f21a9309640 1 -- 172.21.15.77:0/2645464930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a40c9ee0 con 0x7f21a40b4ac0 2024-04-25T01:21:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.416+0000 7f21a9309640 1 -- 172.21.15.77:0/2645464930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21a4068180 con 0x7f21a40b4ac0 2024-04-25T01:21:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.416+0000 7f218b7fe640 1 -- 172.21.15.77:0/2645464930 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2198045070 con 0x7f21a40b4ac0 2024-04-25T01:21:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.416+0000 7f218b7fe640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f219006be30 0x7f219006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.416+0000 7f218b7fe640 1 -- 172.21.15.77:0/2645464930 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f21980ab230 con 0x7f21a40b4ac0 2024-04-25T01:21:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.416+0000 7f21a2ffd640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f219006be30 0x7f219006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.417+0000 7f21a2ffd640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f219006be30 0x7f219006e2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2194007fd0 tx=0x7f2194009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.421+0000 7f218b7fe640 1 -- 172.21.15.77:0/2645464930 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f21a41577c0 con 0x7f21a40b4ac0 2024-04-25T01:21:42.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.592+0000 7f21a9309640 1 -- 172.21.15.77:0/2645464930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f21a40b7bf0 con 0x7f21a40b4ac0 2024-04-25T01:21:42.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.592+0000 7f218b7fe640 1 -- 172.21.15.77:0/2645464930 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f219802ae20 con 0x7f21a40b4ac0 2024-04-25T01:21:42.574 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:21:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 -- 172.21.15.77:0/2645464930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f219006be30 msgr2=0x7f219006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f219006be30 0x7f219006e2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2194007fd0 tx=0x7f2194009040 comp rx=0 tx=0).stop 2024-04-25T01:21:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 -- 172.21.15.77:0/2645464930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21a40b4ac0 msgr2=0x7f21a41e7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21a40b4ac0 0x7f21a41e7390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2198000c00 tx=0x7f21980046c0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 -- 172.21.15.77:0/2645464930 shutdown_connections 2024-04-25T01:21:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f219006be30 0x7f219006e2f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21a40b4ac0 0x7f21a41e7390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21a415f6a0 0x7f21a41e78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 --2- 172.21.15.77:0/2645464930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21a40b2be0 0x7f21a4156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 -- 172.21.15.77:0/2645464930 >> 172.21.15.77:0/2645464930 conn(0x7f21a40b37a0 msgr2=0x7f21a4070210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 -- 172.21.15.77:0/2645464930 shutdown_connections 2024-04-25T01:21:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.594+0000 7f21a9309640 1 -- 172.21.15.77:0/2645464930 wait complete. 2024-04-25T01:21:42.587 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T01:21:42.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.701+0000 7f2c7fb5a640 1 Processor -- start 2024-04-25T01:21:42.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.701+0000 7f2c7fb5a640 1 -- start start 2024-04-25T01:21:42.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7fb5a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c7815b5f0 0x7f2c780711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7fb5a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c78071730 0x7f2c7806e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7fb5a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c78154ea0 con 0x7f2c78154ac0 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7fb5a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f2c7806d540 con 0x7f2c78071730 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7fb5a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2c7806d790 con 0x7f2c7815b5f0 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7e0f7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c78071730 0x7f2c7806e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7d8f6640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2c78154ac0 0x7f2c780cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:40364/0 (socket says 172.21.15.77:40364) 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7d8f6640 1 -- 172.21.15.77:0/363050254 learned_addr learned my addr 172.21.15.77:0/363050254 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7d0f5640 1 --2- 172.21.15.77:0/363050254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c7815b5f0 0x7f2c780711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7e0f7640 1 -- 172.21.15.77:0/363050254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c7815b5f0 msgr2=0x7f2c780711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7e0f7640 1 --2- 172.21.15.77:0/363050254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c7815b5f0 0x7f2c780711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7e0f7640 1 -- 172.21.15.77:0/363050254 >> v1:172.21.15.77:6789/0 conn(0x7f2c78154ac0 legacy=0x7f2c780cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7e0f7640 1 -- 172.21.15.77:0/363050254 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7806da70 con 0x7f2c78071730 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7c8f4640 1 -- 172.21.15.77:0/363050254 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254523649 0 0) 0x7f2c78154ea0 con 0x7f2c78154ac0 2024-04-25T01:21:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.702+0000 7f2c7d0f5640 1 --2- 172.21.15.77:0/363050254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c7815b5f0 0x7f2c780711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:42.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.703+0000 7f2c7e0f7640 1 --2- 172.21.15.77:0/363050254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c78071730 0x7f2c7806e550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2c68009d90 tx=0x7f2c6802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a539de22df689424 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.703+0000 7f2c7c8f4640 1 -- 172.21.15.77:0/363050254 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c68007740 con 0x7f2c78071730 2024-04-25T01:21:42.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.703+0000 7f2c7c8f4640 1 -- 172.21.15.77:0/363050254 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c680078e0 con 0x7f2c78071730 2024-04-25T01:21:42.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.703+0000 7f2c7c8f4640 1 -- 172.21.15.77:0/363050254 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c68007cb0 con 0x7f2c78071730 2024-04-25T01:21:42.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.704+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/363050254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c78071730 msgr2=0x7f2c7806e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.704+0000 7f2c7fb5a640 1 --2- 172.21.15.77:0/363050254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c78071730 0x7f2c7806e550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2c68009d90 tx=0x7f2c6802a660 comp rx=0 tx=0).stop 2024-04-25T01:21:42.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.704+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/363050254 shutdown_connections 2024-04-25T01:21:42.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.704+0000 7f2c7fb5a640 1 --2- 172.21.15.77:0/363050254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c78071730 0x7f2c7806e550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.704+0000 7f2c7fb5a640 1 --2- 172.21.15.77:0/363050254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c7815b5f0 0x7f2c780711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.704+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/363050254 >> 172.21.15.77:0/363050254 conn(0x7f2c780b1820 msgr2=0x7f2c780b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:42.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.704+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/363050254 shutdown_connections 2024-04-25T01:21:42.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.704+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/363050254 wait complete. 2024-04-25T01:21:42.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.705+0000 7f2c7fb5a640 1 Processor -- start 2024-04-25T01:21:42.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.705+0000 7f2c7fb5a640 1 -- start start 2024-04-25T01:21:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.705+0000 7f2c7fb5a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c78071730 0x7f2c78153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7fb5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c78154ac0 0x7f2c781585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7fb5a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c7815b5f0 0x7f2c781558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7fb5a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7806ea90 con 0x7f2c78154ac0 2024-04-25T01:21:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7fb5a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7806e030 con 0x7f2c78071730 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7fb5a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2c7806ec10 con 0x7f2c7815b5f0 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7d0f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c78154ac0 0x7f2c781585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7d0f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c78154ac0 0x7f2c781585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53114/0 (socket says 172.21.15.77:53114) 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7d0f5640 1 -- 172.21.15.77:0/2890965583 learned_addr learned my addr 172.21.15.77:0/2890965583 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7d0f5640 1 -- 172.21.15.77:0/2890965583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c7815b5f0 msgr2=0x7f2c781558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7d0f5640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c7815b5f0 0x7f2c781558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7d0f5640 1 -- 172.21.15.77:0/2890965583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c78071730 msgr2=0x7f2c78153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7d0f5640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c78071730 0x7f2c78153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7d0f5640 1 -- 172.21.15.77:0/2890965583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c78159020 con 0x7f2c78154ac0 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7d0f5640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c78154ac0 0x7f2c781585d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00c7f0 tx=0x7f2c6c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c767fc640 1 -- 172.21.15.77:0/2890965583 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c015070 con 0x7f2c78154ac0 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c767fc640 1 -- 172.21.15.77:0/2890965583 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c6c007820 con 0x7f2c78154ac0 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/2890965583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c78159300 con 0x7f2c78154ac0 2024-04-25T01:21:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c767fc640 1 -- 172.21.15.77:0/2890965583 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c007bd0 con 0x7f2c78154ac0 2024-04-25T01:21:42.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.706+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/2890965583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c781598c0 con 0x7f2c78154ac0 2024-04-25T01:21:42.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.707+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/2890965583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c78068fb0 con 0x7f2c78154ac0 2024-04-25T01:21:42.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.708+0000 7f2c767fc640 1 -- 172.21.15.77:0/2890965583 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2c6c018070 con 0x7f2c78154ac0 2024-04-25T01:21:42.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.708+0000 7f2c767fc640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c6406be60 0x7f2c6406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.708+0000 7f2c7d8f6640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c6406be60 0x7f2c6406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.708+0000 7f2c767fc640 1 -- 172.21.15.77:0/2890965583 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f2c6c08ab80 con 0x7f2c78154ac0 2024-04-25T01:21:42.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.708+0000 7f2c7d8f6640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c6406be60 0x7f2c6406e320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c70007fd0 tx=0x7f2c70007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.712+0000 7f2c767fc640 1 -- 172.21.15.77:0/2890965583 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2c6c059910 con 0x7f2c78154ac0 2024-04-25T01:21:42.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.888+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/2890965583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2c7815da70 con 0x7f2c78154ac0 2024-04-25T01:21:42.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.888+0000 7f2c767fc640 1 -- 172.21.15.77:0/2890965583 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f2c6c059210 con 0x7f2c78154ac0 2024-04-25T01:21:42.871 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:42.871 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-04-25T01:21:42.871 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:21:42.871 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-04-25T01:21:42.871 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-04-25T01:21:42.871 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:21:42.871 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-04-25T01:21:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.890+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/2890965583 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c6406be60 msgr2=0x7f2c6406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.890+0000 7f2c7fb5a640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c6406be60 0x7f2c6406e320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c70007fd0 tx=0x7f2c70007920 comp rx=0 tx=0).stop 2024-04-25T01:21:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.890+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/2890965583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c78154ac0 msgr2=0x7f2c781585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.890+0000 7f2c7fb5a640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c78154ac0 0x7f2c781585d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00c7f0 tx=0x7f2c6c00ccb0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.890+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/2890965583 shutdown_connections 2024-04-25T01:21:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.890+0000 7f2c7fb5a640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c6406be60 0x7f2c6406e320 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.890+0000 7f2c7fb5a640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c78154ac0 0x7f2c781585d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.890+0000 7f2c7fb5a640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c7815b5f0 0x7f2c781558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.890+0000 7f2c7fb5a640 1 --2- 172.21.15.77:0/2890965583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c78071730 0x7f2c78153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.890+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/2890965583 >> 172.21.15.77:0/2890965583 conn(0x7f2c780b1820 msgr2=0x7f2c780b3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.891+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/2890965583 shutdown_connections 2024-04-25T01:21:42.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:42.891+0000 7f2c7fb5a640 1 -- 172.21.15.77:0/2890965583 wait complete. 2024-04-25T01:21:42.883 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T01:21:42.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T01:21:42.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.001+0000 7fe04dfd5640 1 Processor -- start 2024-04-25T01:21:42.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.001+0000 7fe04dfd5640 1 -- start start 2024-04-25T01:21:42.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.001+0000 7fe04dfd5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe048070700 0x7fe0480af160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.001+0000 7fe04dfd5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe04806c230 0x7fe04806c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.001+0000 7fe04dfd5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe04806f940 con 0x7fe04815e240 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.001+0000 7fe04dfd5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fe048070b70 con 0x7fe048070700 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.001+0000 7fe04dfd5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe04806fc20 con 0x7fe04806c230 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.002+0000 7fe046ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe048070700 0x7fe0480af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.002+0000 7fe046ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe048070700 0x7fe0480af160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53128/0 (socket says 172.21.15.77:53128) 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.002+0000 7fe046ffd640 1 -- 172.21.15.77:0/3487881900 learned_addr learned my addr 172.21.15.77:0/3487881900 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.002+0000 7fe047fff640 1 --2- 172.21.15.77:0/3487881900 >> v2:172.21.15.121:3300/0 conn(0x7fe04806c230 0x7fe04806c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.002+0000 7fe046ffd640 1 -- 172.21.15.77:0/3487881900 >> v2:172.21.15.121:3300/0 conn(0x7fe04806c230 msgr2=0x7fe04806c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.002+0000 7fe046ffd640 1 --2- 172.21.15.77:0/3487881900 >> v2:172.21.15.121:3300/0 conn(0x7fe04806c230 0x7fe04806c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.002+0000 7fe046ffd640 1 -- 172.21.15.77:0/3487881900 >> v1:172.21.15.121:6789/0 conn(0x7fe04815e240 legacy=0x7fe0480aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.002+0000 7fe046ffd640 1 -- 172.21.15.77:0/3487881900 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe04806ff00 con 0x7fe048070700 2024-04-25T01:21:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.002+0000 7fe046ffd640 1 --2- 172.21.15.77:0/3487881900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe048070700 0x7fe0480af160 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe0480d1c90 tx=0x7fe03c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40e9c35662c0262 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.003+0000 7fe0467fc640 1 -- 172.21.15.77:0/3487881900 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c037070 con 0x7fe048070700 2024-04-25T01:21:42.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.003+0000 7fe0467fc640 1 -- 172.21.15.77:0/3487881900 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe03c02ae50 con 0x7fe048070700 2024-04-25T01:21:42.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.003+0000 7fe0467fc640 1 -- 172.21.15.77:0/3487881900 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c032860 con 0x7fe048070700 2024-04-25T01:21:42.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.003+0000 7fe04dfd5640 1 -- 172.21.15.77:0/3487881900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe048070700 msgr2=0x7fe0480af160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.003+0000 7fe04dfd5640 1 --2- 172.21.15.77:0/3487881900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe048070700 0x7fe0480af160 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe0480d1c90 tx=0x7fe03c02a300 comp rx=0 tx=0).stop 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.003+0000 7fe04dfd5640 1 -- 172.21.15.77:0/3487881900 shutdown_connections 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.003+0000 7fe04dfd5640 1 --2- 172.21.15.77:0/3487881900 >> v2:172.21.15.121:3300/0 conn(0x7fe04806c230 0x7fe04806c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.003+0000 7fe04dfd5640 1 --2- 172.21.15.77:0/3487881900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe048070700 0x7fe0480af160 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.003+0000 7fe04dfd5640 1 -- 172.21.15.77:0/3487881900 >> 172.21.15.77:0/3487881900 conn(0x7fe0480d1d00 msgr2=0x7fe0480d2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.003+0000 7fe04dfd5640 1 -- 172.21.15.77:0/3487881900 shutdown_connections 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.003+0000 7fe04dfd5640 1 -- 172.21.15.77:0/3487881900 wait complete. 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.004+0000 7fe04dfd5640 1 Processor -- start 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.005+0000 7fe04dfd5640 1 -- start start 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.005+0000 7fe04dfd5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe04806c230 0x7fe0481f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.005+0000 7fe04dfd5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe048070700 0x7fe0481f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.005+0000 7fe04dfd5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe04815e240 0x7fe0480b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.005+0000 7fe04dfd5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe0480d0fc0 con 0x7fe048070700 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.005+0000 7fe04dfd5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe0480af6a0 con 0x7fe04815e240 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.005+0000 7fe04dfd5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe04806cbc0 con 0x7fe04806c230 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.005+0000 7fe046ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe048070700 0x7fe0481f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.005+0000 7fe046ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe048070700 0x7fe0481f7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53138/0 (socket says 172.21.15.77:53138) 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.005+0000 7fe046ffd640 1 -- 172.21.15.77:0/535260631 learned_addr learned my addr 172.21.15.77:0/535260631 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe046ffd640 1 -- 172.21.15.77:0/535260631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe04806c230 msgr2=0x7fe0481f47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe046ffd640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe04806c230 0x7fe0481f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe046ffd640 1 -- 172.21.15.77:0/535260631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe04815e240 msgr2=0x7fe0480b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe047fff640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe04815e240 0x7fe0480b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe046ffd640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe04815e240 0x7fe0480b0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe046ffd640 1 -- 172.21.15.77:0/535260631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0480c68b0 con 0x7fe048070700 2024-04-25T01:21:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe046ffd640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe048070700 0x7fe0481f7270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe03c02a340 tx=0x7fe03c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe02ffff640 1 -- 172.21.15.77:0/535260631 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c041070 con 0x7fe048070700 2024-04-25T01:21:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe02ffff640 1 -- 172.21.15.77:0/535260631 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe03c03c670 con 0x7fe048070700 2024-04-25T01:21:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe04dfd5640 1 -- 172.21.15.77:0/535260631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0481f7a50 con 0x7fe048070700 2024-04-25T01:21:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe02ffff640 1 -- 172.21.15.77:0/535260631 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c037040 con 0x7fe048070700 2024-04-25T01:21:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.006+0000 7fe04dfd5640 1 -- 172.21.15.77:0/535260631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0481f8010 con 0x7fe048070700 2024-04-25T01:21:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.007+0000 7fe04dfd5640 1 -- 172.21.15.77:0/535260631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe048068180 con 0x7fe048070700 2024-04-25T01:21:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.007+0000 7fe02ffff640 1 -- 172.21.15.77:0/535260631 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe03c044070 con 0x7fe048070700 2024-04-25T01:21:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.008+0000 7fe02ffff640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe03406be30 0x7fe03406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.008+0000 7fe02ffff640 1 -- 172.21.15.77:0/535260631 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fe03c0ab5d0 con 0x7fe048070700 2024-04-25T01:21:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.008+0000 7fe0477fe640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe03406be30 0x7fe03406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.008+0000 7fe0477fe640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe03406be30 0x7fe03406e2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe038000f60 tx=0x7fe0380079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:42.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.012+0000 7fe02ffff640 1 -- 172.21.15.77:0/535260631 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe03c07a370 con 0x7fe048070700 2024-04-25T01:21:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:43.188+0000 7fe04dfd5640 1 -- 172.21.15.77:0/535260631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fe04815d970 con 0x7fe048070700 2024-04-25T01:21:44.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.055+0000 7fe02ffff640 1 -- 172.21.15.77:0/535260631 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7fe03c080020 con 0x7fe048070700 2024-04-25T01:21:44.038 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-04-25T01:21:44.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.058+0000 7fe04dfd5640 1 -- 172.21.15.77:0/535260631 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe03406be30 msgr2=0x7fe03406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:44.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.058+0000 7fe04dfd5640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe03406be30 0x7fe03406e2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe038000f60 tx=0x7fe0380079d0 comp rx=0 tx=0).stop 2024-04-25T01:21:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.059+0000 7fe04dfd5640 1 -- 172.21.15.77:0/535260631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe048070700 msgr2=0x7fe0481f7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.059+0000 7fe04dfd5640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe048070700 0x7fe0481f7270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe03c02a340 tx=0x7fe03c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:21:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.059+0000 7fe04dfd5640 1 -- 172.21.15.77:0/535260631 shutdown_connections 2024-04-25T01:21:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.059+0000 7fe04dfd5640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe03406be30 0x7fe03406e2f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.059+0000 7fe04dfd5640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe048070700 0x7fe0481f7270 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.059+0000 7fe04dfd5640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe04815e240 0x7fe0480b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.059+0000 7fe04dfd5640 1 --2- 172.21.15.77:0/535260631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe04806c230 0x7fe0481f47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.059+0000 7fe04dfd5640 1 -- 172.21.15.77:0/535260631 >> 172.21.15.77:0/535260631 conn(0x7fe0480d1d00 msgr2=0x7fe0480af820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.060+0000 7fe04dfd5640 1 -- 172.21.15.77:0/535260631 shutdown_connections 2024-04-25T01:21:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.060+0000 7fe04dfd5640 1 -- 172.21.15.77:0/535260631 wait complete. 2024-04-25T01:21:44.056 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-25T01:21:44.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.179+0000 7fb60ed7e640 1 Processor -- start 2024-04-25T01:21:44.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.179+0000 7fb60ed7e640 1 -- start start 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.180+0000 7fb60ed7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb60815c080 0x7fb6080b5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.180+0000 7fb60ed7e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb60806b830 0x7fb608066ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.180+0000 7fb60ed7e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6080cdc70 con 0x7fb6080cd7f0 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.180+0000 7fb60ed7e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb6080b5220 con 0x7fb60806b830 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.180+0000 7fb60ed7e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb6080b5500 con 0x7fb60815c080 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.180+0000 7fb60d31b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb60806b830 0x7fb608066ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.180+0000 7fb60cb1a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb60815c080 0x7fb6080b5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.180+0000 7fb607fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb6080cd7f0 0x7fb60806b120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:49972/0 (socket says 172.21.15.77:49972) 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.180+0000 7fb607fff640 1 -- 172.21.15.77:0/1061538636 learned_addr learned my addr 172.21.15.77:0/1061538636 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.181+0000 7fb60cb1a640 1 -- 172.21.15.77:0/1061538636 >> v2:172.21.15.121:3300/0 conn(0x7fb60806b830 msgr2=0x7fb608066ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.181+0000 7fb60cb1a640 1 --2- 172.21.15.77:0/1061538636 >> v2:172.21.15.121:3300/0 conn(0x7fb60806b830 0x7fb608066ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.181+0000 7fb60cb1a640 1 -- 172.21.15.77:0/1061538636 >> v1:172.21.15.121:6789/0 conn(0x7fb6080cd7f0 legacy=0x7fb60806b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.181+0000 7fb60cb1a640 1 -- 172.21.15.77:0/1061538636 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6080b57e0 con 0x7fb60815c080 2024-04-25T01:21:44.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.181+0000 7fb60d31b640 1 --2- 172.21.15.77:0/1061538636 >> v2:172.21.15.121:3300/0 conn(0x7fb60806b830 0x7fb608066ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.182+0000 7fb60cb1a640 1 --2- 172.21.15.77:0/1061538636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb60815c080 0x7fb6080b5fe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5f80097a0 tx=0x7fb5f802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46ef7b208d3c8841 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.182+0000 7fb6077fe640 1 -- 172.21.15.77:0/1061538636 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8037070 con 0x7fb60815c080 2024-04-25T01:21:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.182+0000 7fb6077fe640 1 -- 172.21.15.77:0/1061538636 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5f8032440 con 0x7fb60815c080 2024-04-25T01:21:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.182+0000 7fb6077fe640 1 -- 172.21.15.77:0/1061538636 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8032810 con 0x7fb60815c080 2024-04-25T01:21:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.182+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1061538636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb60815c080 msgr2=0x7fb6080b5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.182+0000 7fb60ed7e640 1 --2- 172.21.15.77:0/1061538636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb60815c080 0x7fb6080b5fe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5f80097a0 tx=0x7fb5f802a180 comp rx=0 tx=0).stop 2024-04-25T01:21:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.183+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1061538636 shutdown_connections 2024-04-25T01:21:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.183+0000 7fb60ed7e640 1 --2- 172.21.15.77:0/1061538636 >> v2:172.21.15.121:3300/0 conn(0x7fb60806b830 0x7fb608066ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.183+0000 7fb60ed7e640 1 --2- 172.21.15.77:0/1061538636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb60815c080 0x7fb6080b5fe0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:44.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.183+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1061538636 >> 172.21.15.77:0/1061538636 conn(0x7fb6080b1f00 msgr2=0x7fb6080b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:44.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.183+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1061538636 shutdown_connections 2024-04-25T01:21:44.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.183+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1061538636 wait complete. 2024-04-25T01:21:44.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.184+0000 7fb60ed7e640 1 Processor -- start 2024-04-25T01:21:44.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.184+0000 7fb60ed7e640 1 -- start start 2024-04-25T01:21:44.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb60ed7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb60806b830 0x7fb60814a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:44.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb60ed7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6080cd7f0 0x7fb608159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:44.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb60ed7e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb60815c080 0x7fb608159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb60ed7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb6080ce560 con 0x7fb6080cd7f0 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb60ed7e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb60815c460 con 0x7fb60815c080 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb60ed7e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb6080ce6e0 con 0x7fb60806b830 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb607fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6080cd7f0 0x7fb608159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb607fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6080cd7f0 0x7fb608159a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53140/0 (socket says 172.21.15.77:53140) 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb607fff640 1 -- 172.21.15.77:0/1119274654 learned_addr learned my addr 172.21.15.77:0/1119274654 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb60d31b640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb60815c080 0x7fb608159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb60cb1a640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb60806b830 0x7fb60814a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb607fff640 1 -- 172.21.15.77:0/1119274654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb60806b830 msgr2=0x7fb60814a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb607fff640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb60806b830 0x7fb60814a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb607fff640 1 -- 172.21.15.77:0/1119274654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb60815c080 msgr2=0x7fb608159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb607fff640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb60815c080 0x7fb608159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb607fff640 1 -- 172.21.15.77:0/1119274654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6080c9dc0 con 0x7fb6080cd7f0 2024-04-25T01:21:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb607fff640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6080cd7f0 0x7fb608159a60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00c430 tx=0x7fb5fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb6057fa640 1 -- 172.21.15.77:0/1119274654 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc017070 con 0x7fb6080cd7f0 2024-04-25T01:21:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1119274654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6080c3f00 con 0x7fb6080cd7f0 2024-04-25T01:21:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb6057fa640 1 -- 172.21.15.77:0/1119274654 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5fc005ce0 con 0x7fb6080cd7f0 2024-04-25T01:21:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb6057fa640 1 -- 172.21.15.77:0/1119274654 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0072f0 con 0x7fb6080cd7f0 2024-04-25T01:21:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.185+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1119274654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6080c44c0 con 0x7fb6080cd7f0 2024-04-25T01:21:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.186+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1119274654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb608068fb0 con 0x7fb6080cd7f0 2024-04-25T01:21:44.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.187+0000 7fb6057fa640 1 -- 172.21.15.77:0/1119274654 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb5fc01a070 con 0x7fb6080cd7f0 2024-04-25T01:21:44.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.187+0000 7fb6057fa640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb5f406be30 0x7fb5f406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:44.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.187+0000 7fb6057fa640 1 -- 172.21.15.77:0/1119274654 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7fb5fc08a0e0 con 0x7fb6080cd7f0 2024-04-25T01:21:44.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.187+0000 7fb60cb1a640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb5f406be30 0x7fb5f406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:44.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.188+0000 7fb60cb1a640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb5f406be30 0x7fb5f406e2f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8034000 tx=0x7fb5f802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:44.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.192+0000 7fb6057fa640 1 -- 172.21.15.77:0/1119274654 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb5fc058510 con 0x7fb6080cd7f0 2024-04-25T01:21:44.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:44.366+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1119274654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fb6080cfeb0 con 0x7fb6080cd7f0 2024-04-25T01:21:45.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.057+0000 7fb6057fa640 1 -- 172.21.15.77:0/1119274654 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7fb5fc05c810 con 0x7fb6080cd7f0 2024-04-25T01:21:45.039 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-04-25T01:21:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.059+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1119274654 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb5f406be30 msgr2=0x7fb5f406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.059+0000 7fb60ed7e640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb5f406be30 0x7fb5f406e2f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8034000 tx=0x7fb5f802e000 comp rx=0 tx=0).stop 2024-04-25T01:21:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.059+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1119274654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6080cd7f0 msgr2=0x7fb608159a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.059+0000 7fb60ed7e640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6080cd7f0 0x7fb608159a60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00c430 tx=0x7fb5fc00c8f0 comp rx=0 tx=0).stop 2024-04-25T01:21:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.060+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1119274654 shutdown_connections 2024-04-25T01:21:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.060+0000 7fb60ed7e640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb5f406be30 0x7fb5f406e2f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.060+0000 7fb60ed7e640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6080cd7f0 0x7fb608159a60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.060+0000 7fb60ed7e640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb60815c080 0x7fb608159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.060+0000 7fb60ed7e640 1 --2- 172.21.15.77:0/1119274654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb60806b830 0x7fb60814a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.060+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1119274654 >> 172.21.15.77:0/1119274654 conn(0x7fb6080b1f00 msgr2=0x7fb6080b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.060+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1119274654 shutdown_connections 2024-04-25T01:21:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.060+0000 7fb60ed7e640 1 -- 172.21.15.77:0/1119274654 wait complete. 2024-04-25T01:21:45.064 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T01:21:45.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.183+0000 7f8aa1142640 1 Processor -- start 2024-04-25T01:21:45.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.183+0000 7f8aa1142640 1 -- start start 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8aa1142640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8a9c0d23d0 0x7f8a9c0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8aa1142640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a9c0d27b0 0x7f8a9c0d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8aa1142640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a9c0cea40 con 0x7f8a9c0b9380 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8aa1142640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f8a9c0cece0 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8aa1142640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8a9c0cefc0 con 0x7f8a9c0d23d0 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8a93fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a9c0d27b0 0x7f8a9c0d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8a93fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a9c0d27b0 0x7f8a9c0d0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53154/0 (socket says 172.21.15.77:53154) 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8a93fff640 1 -- 172.21.15.77:0/3737084947 learned_addr learned my addr 172.21.15.77:0/3737084947 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8a93fff640 1 -- 172.21.15.77:0/3737084947 >> v2:172.21.15.121:3300/0 conn(0x7f8a9c0d23d0 msgr2=0x7f8a9c0d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8a93fff640 1 --2- 172.21.15.77:0/3737084947 >> v2:172.21.15.121:3300/0 conn(0x7f8a9c0d23d0 0x7f8a9c0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8a93fff640 1 -- 172.21.15.77:0/3737084947 >> v1:172.21.15.121:6789/0 conn(0x7f8a9c0b9380 legacy=0x7f8a9c0b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.184+0000 7f8a93fff640 1 -- 172.21.15.77:0/3737084947 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9c0cf2a0 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8a93fff640 1 --2- 172.21.15.77:0/3737084947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27b0 0x7f8a9c0d0ee0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c067900 tx=0x7f8a8402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c888f4980cd5cd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:45.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8a9a59c640 1 -- 172.21.15.77:0/3737084947 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a84037070 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8a9a59c640 1 -- 172.21.15.77:0/3737084947 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a84032440 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8a9a59c640 1 -- 172.21.15.77:0/3737084947 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a84032810 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8aa1142640 1 -- 172.21.15.77:0/3737084947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27b0 msgr2=0x7f8a9c0d0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:45.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8aa1142640 1 --2- 172.21.15.77:0/3737084947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27b0 0x7f8a9c0d0ee0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c067900 tx=0x7f8a8402a2c0 comp rx=0 tx=0).stop 2024-04-25T01:21:45.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8aa1142640 1 -- 172.21.15.77:0/3737084947 shutdown_connections 2024-04-25T01:21:45.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8aa1142640 1 --2- 172.21.15.77:0/3737084947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27b0 0x7f8a9c0d0ee0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:45.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8aa1142640 1 --2- 172.21.15.77:0/3737084947 >> v2:172.21.15.121:3300/0 conn(0x7f8a9c0d23d0 0x7f8a9c0d4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:45.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8aa1142640 1 -- 172.21.15.77:0/3737084947 >> 172.21.15.77:0/3737084947 conn(0x7f8a9c0d3e30 msgr2=0x7f8a9c0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:45.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8aa1142640 1 -- 172.21.15.77:0/3737084947 shutdown_connections 2024-04-25T01:21:45.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.185+0000 7f8aa1142640 1 -- 172.21.15.77:0/3737084947 wait complete. 2024-04-25T01:21:45.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.187+0000 7f8aa1142640 1 Processor -- start 2024-04-25T01:21:45.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.187+0000 7f8aa1142640 1 -- start start 2024-04-25T01:21:45.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.187+0000 7f8aa1142640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a9c0b9380 0x7f8a9c1f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:45.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8aa1142640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a9c0d23d0 0x7f8a9c1f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:45.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8aa1142640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27b0 0x7f8a9c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:45.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8aa1142640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c0cfd80 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8aa1142640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c0cf860 con 0x7f8a9c0d23d0 2024-04-25T01:21:45.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8aa1142640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8a9c0b6c80 con 0x7f8a9c0b9380 2024-04-25T01:21:45.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a9b59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27b0 0x7f8a9c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:45.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a9b59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27b0 0x7f8a9c15ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53168/0 (socket says 172.21.15.77:53168) 2024-04-25T01:21:45.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a9b59e640 1 -- 172.21.15.77:0/2440583241 learned_addr learned my addr 172.21.15.77:0/2440583241 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:45.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a93fff640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a9c0d23d0 0x7f8a9c1f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a9ad9d640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a9c0b9380 0x7f8a9c1f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a9b59e640 1 -- 172.21.15.77:0/2440583241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a9c0b9380 msgr2=0x7f8a9c1f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a9b59e640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a9c0b9380 0x7f8a9c1f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a9b59e640 1 -- 172.21.15.77:0/2440583241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a9c0d23d0 msgr2=0x7f8a9c1f49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a9b59e640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a9c0d23d0 0x7f8a9c1f49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a9b59e640 1 -- 172.21.15.77:0/2440583241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9c15d430 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a93fff640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a9c0d23d0 0x7f8a9c1f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a9ad9d640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a9c0b9380 0x7f8a9c1f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a9b59e640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27b0 0x7f8a9c15ccb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8a9400c780 tx=0x7f8a9400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a937fe640 1 -- 172.21.15.77:0/2440583241 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94017070 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a937fe640 1 -- 172.21.15.77:0/2440583241 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a94003da0 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8aa1142640 1 -- 172.21.15.77:0/2440583241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a9c15d710 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8a937fe640 1 -- 172.21.15.77:0/2440583241 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a940052b0 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.188+0000 7f8aa1142640 1 -- 172.21.15.77:0/2440583241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a9c15b4e0 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.189+0000 7f8aa1142640 1 -- 172.21.15.77:0/2440583241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a9c15dc30 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.190+0000 7f8a937fe640 1 -- 172.21.15.77:0/2440583241 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8a9401a070 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.190+0000 7f8a937fe640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8a7c06be30 0x7f8a7c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:45.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.190+0000 7f8a9ad9d640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8a7c06be30 0x7f8a7c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:45.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.190+0000 7f8a937fe640 1 -- 172.21.15.77:0/2440583241 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7f8a9408a4a0 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.191+0000 7f8a9ad9d640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8a7c06be30 0x7f8a7c06e2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c006fd0 tx=0x7f8a8c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:45.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.195+0000 7f8a937fe640 1 -- 172.21.15.77:0/2440583241 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8a9c15dc30 con 0x7f8a9c0d27b0 2024-04-25T01:21:45.373 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:21:45.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:21:45.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:45.406+0000 7f8aa1142640 1 -- 172.21.15.77:0/2440583241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f8a9c068fb0 con 0x7f8a9c0d27b0 2024-04-25T01:21:46.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.070+0000 7f8a937fe640 1 -- 172.21.15.77:0/2440583241 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f8a94058eb0 con 0x7f8a9c0d27b0 2024-04-25T01:21:46.052 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T01:21:46.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.074+0000 7f8aa1142640 1 -- 172.21.15.77:0/2440583241 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8a7c06be30 msgr2=0x7f8a7c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:46.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.074+0000 7f8aa1142640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8a7c06be30 0x7f8a7c06e2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c006fd0 tx=0x7f8a8c008040 comp rx=0 tx=0).stop 2024-04-25T01:21:46.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.074+0000 7f8aa1142640 1 -- 172.21.15.77:0/2440583241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27b0 msgr2=0x7f8a9c15ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:46.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.074+0000 7f8aa1142640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27b0 0x7f8a9c15ccb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8a9400c780 tx=0x7f8a9400cc40 comp rx=0 tx=0).stop 2024-04-25T01:21:46.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.075+0000 7f8aa1142640 1 -- 172.21.15.77:0/2440583241 shutdown_connections 2024-04-25T01:21:46.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.075+0000 7f8aa1142640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8a7c06be30 0x7f8a7c06e2f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.075+0000 7f8aa1142640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27b0 0x7f8a9c15ccb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.075+0000 7f8aa1142640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a9c0d23d0 0x7f8a9c1f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.075+0000 7f8aa1142640 1 --2- 172.21.15.77:0/2440583241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a9c0b9380 0x7f8a9c1f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.075+0000 7f8aa1142640 1 -- 172.21.15.77:0/2440583241 >> 172.21.15.77:0/2440583241 conn(0x7f8a9c0d3e30 msgr2=0x7f8a9c074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:46.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.075+0000 7f8aa1142640 1 -- 172.21.15.77:0/2440583241 shutdown_connections 2024-04-25T01:21:46.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.075+0000 7f8aa1142640 1 -- 172.21.15.77:0/2440583241 wait complete. 2024-04-25T01:21:46.084 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-25T01:21:46.084 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-25T01:21:46.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.206+0000 7f50fdeb3640 1 Processor -- start 2024-04-25T01:21:46.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.206+0000 7f50fdeb3640 1 -- start start 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.207+0000 7f50fdeb3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f50f8161810 0x7f50f80cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.207+0000 7f50fdeb3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f50f80d28a0 0x7f50f80b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.207+0000 7f50fdeb3640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f80d0700 con 0x7f50f80cf9e0 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.207+0000 7f50fdeb3640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f50f80d04b0 con 0x7f50f8161810 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.207+0000 7f50fdeb3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f50f80d09e0 con 0x7f50f80d28a0 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.207+0000 7f50f77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f50f8161810 0x7f50f80cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.207+0000 7f50f6ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f50f80cf9e0 0x7f50f80d2190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:40376/0 (socket says 172.21.15.77:40376) 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.207+0000 7f50f6ffd640 1 -- 172.21.15.77:0/1109228355 learned_addr learned my addr 172.21.15.77:0/1109228355 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.208+0000 7f50f7fff640 1 --2- 172.21.15.77:0/1109228355 >> v2:172.21.15.121:3300/0 conn(0x7f50f80d28a0 0x7f50f80b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.208+0000 7f50f77fe640 1 -- 172.21.15.77:0/1109228355 >> v2:172.21.15.121:3300/0 conn(0x7f50f80d28a0 msgr2=0x7f50f80b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.208+0000 7f50f77fe640 1 --2- 172.21.15.77:0/1109228355 >> v2:172.21.15.121:3300/0 conn(0x7f50f80d28a0 0x7f50f80b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.208+0000 7f50f77fe640 1 -- 172.21.15.77:0/1109228355 >> v1:172.21.15.77:6789/0 conn(0x7f50f80cf9e0 legacy=0x7f50f80d2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:46.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.208+0000 7f50f77fe640 1 -- 172.21.15.77:0/1109228355 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f80d0cc0 con 0x7f50f8161810 2024-04-25T01:21:46.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.208+0000 7f50f67fc640 1 -- 172.21.15.77:0/1109228355 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2073403613 0 0) 0x7f50f80d0700 con 0x7f50f80cf9e0 2024-04-25T01:21:46.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.208+0000 7f50f77fe640 1 --2- 172.21.15.77:0/1109228355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50f8161810 0x7f50f80cf4a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50e80097a0 tx=0x7f50e802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fd4b9e98eaa195 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:46.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.208+0000 7f50f67fc640 1 -- 172.21.15.77:0/1109228355 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e8037070 con 0x7f50f8161810 2024-04-25T01:21:46.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.208+0000 7f50f67fc640 1 -- 172.21.15.77:0/1109228355 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50e8032440 con 0x7f50f8161810 2024-04-25T01:21:46.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.208+0000 7f50f67fc640 1 -- 172.21.15.77:0/1109228355 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e8032810 con 0x7f50f8161810 2024-04-25T01:21:46.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.209+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1109228355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50f8161810 msgr2=0x7f50f80cf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:46.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.209+0000 7f50fdeb3640 1 --2- 172.21.15.77:0/1109228355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50f8161810 0x7f50f80cf4a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50e80097a0 tx=0x7f50e802a1e0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.209+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1109228355 shutdown_connections 2024-04-25T01:21:46.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.209+0000 7f50fdeb3640 1 --2- 172.21.15.77:0/1109228355 >> v2:172.21.15.121:3300/0 conn(0x7f50f80d28a0 0x7f50f80b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.209+0000 7f50fdeb3640 1 --2- 172.21.15.77:0/1109228355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50f8161810 0x7f50f80cf4a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.209+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1109228355 >> 172.21.15.77:0/1109228355 conn(0x7f50f80d3e80 msgr2=0x7f50f80d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:46.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.209+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1109228355 shutdown_connections 2024-04-25T01:21:46.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.209+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1109228355 wait complete. 2024-04-25T01:21:46.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.210+0000 7f50fdeb3640 1 Processor -- start 2024-04-25T01:21:46.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.210+0000 7f50fdeb3640 1 -- start start 2024-04-25T01:21:46.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50fdeb3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50f80cf9e0 0x7f50f8160740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:46.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50fdeb3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50f80d28a0 0x7f50f815ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:46.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50fdeb3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f50f8161810 0x7f50f80c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:46.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50fdeb3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f50f80d1f70 con 0x7f50f80d28a0 2024-04-25T01:21:46.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50fdeb3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f50f80d1df0 con 0x7f50f80cf9e0 2024-04-25T01:21:46.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50fdeb3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f50f80d14c0 con 0x7f50f8161810 2024-04-25T01:21:46.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50f80d28a0 0x7f50f815ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:46.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50f80d28a0 0x7f50f815ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53178/0 (socket says 172.21.15.77:53178) 2024-04-25T01:21:46.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f6ffd640 1 -- 172.21.15.77:0/1619832334 learned_addr learned my addr 172.21.15.77:0/1619832334 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:46.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f77fe640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50f80cf9e0 0x7f50f8160740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f7fff640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f50f8161810 0x7f50f80c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f6ffd640 1 -- 172.21.15.77:0/1619832334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f50f8161810 msgr2=0x7f50f80c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f6ffd640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f50f8161810 0x7f50f80c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f6ffd640 1 -- 172.21.15.77:0/1619832334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50f80cf9e0 msgr2=0x7f50f8160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f6ffd640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50f80cf9e0 0x7f50f8160740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f6ffd640 1 -- 172.21.15.77:0/1619832334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f80b84a0 con 0x7f50f80d28a0 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f7fff640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f50f8161810 0x7f50f80c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f77fe640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50f80cf9e0 0x7f50f8160740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50f6ffd640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50f80d28a0 0x7f50f815ec10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00c4d0 tx=0x7f50ec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.211+0000 7f50dffff640 1 -- 172.21.15.77:0/1619832334 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec018070 con 0x7f50f80d28a0 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.212+0000 7f50dffff640 1 -- 172.21.15.77:0/1619832334 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50ec007500 con 0x7f50f80d28a0 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.212+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1619832334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50f80b8a30 con 0x7f50f80d28a0 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.212+0000 7f50dffff640 1 -- 172.21.15.77:0/1619832334 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec004a20 con 0x7f50f80d28a0 2024-04-25T01:21:46.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.212+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1619832334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50f80b8ff0 con 0x7f50f80d28a0 2024-04-25T01:21:46.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.213+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1619832334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50f8068fb0 con 0x7f50f80d28a0 2024-04-25T01:21:46.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.213+0000 7f50dffff640 1 -- 172.21.15.77:0/1619832334 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f50ec01b070 con 0x7f50f80d28a0 2024-04-25T01:21:46.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.213+0000 7f50dffff640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f50e406be60 0x7f50e406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:46.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.214+0000 7f50f77fe640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f50e406be60 0x7f50e406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:46.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.214+0000 7f50dffff640 1 -- 172.21.15.77:0/1619832334 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f50ec08a0a0 con 0x7f50f80d28a0 2024-04-25T01:21:46.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.214+0000 7f50f77fe640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f50e406be60 0x7f50e406e320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f50e8034000 tx=0x7f50e802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:46.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.218+0000 7f50dffff640 1 -- 172.21.15.77:0/1619832334 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f50ec058ae0 con 0x7f50f80d28a0 2024-04-25T01:21:46.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:46.400+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1619832334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f50f808d660 con 0x7f50f80d28a0 2024-04-25T01:21:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.071+0000 7f50dffff640 1 -- 172.21.15.77:0/1619832334 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7f50ec0583e0 con 0x7f50f80d28a0 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1619832334 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f50e406be60 msgr2=0x7f50e406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f50e406be60 0x7f50e406e320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f50e8034000 tx=0x7f50e802e000 comp rx=0 tx=0).stop 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1619832334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50f80d28a0 msgr2=0x7f50f815ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50f80d28a0 0x7f50f815ec10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00c4d0 tx=0x7f50ec00c990 comp rx=0 tx=0).stop 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1619832334 shutdown_connections 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f50e406be60 0x7f50e406e320 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50f80d28a0 0x7f50f815ec10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f50f8161810 0x7f50f80c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 --2- 172.21.15.77:0/1619832334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50f80cf9e0 0x7f50f8160740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1619832334 >> 172.21.15.77:0/1619832334 conn(0x7f50f80d3e80 msgr2=0x7f50f80d4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1619832334 shutdown_connections 2024-04-25T01:21:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.074+0000 7f50fdeb3640 1 -- 172.21.15.77:0/1619832334 wait complete. 2024-04-25T01:21:47.068 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T01:21:47.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.193+0000 7fd8a9e38640 1 Processor -- start 2024-04-25T01:21:47.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.193+0000 7fd8a9e38640 1 -- start start 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.194+0000 7fd8a9e38640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd8a4145730 0x7fd8a415f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.194+0000 7fd8a9e38640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a406b830 0x7fd8a4070180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.194+0000 7fd8a9e38640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a415fee0 con 0x7fd8a40b3ac0 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.194+0000 7fd8a9e38640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd8a40706c0 con 0x7fd8a4145730 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.194+0000 7fd8a9e38640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a4145b10 con 0x7fd8a406b830 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.194+0000 7fd89affd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd8a40b3ac0 0x7fd8a406b120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53814/0 (socket says 172.21.15.77:53814) 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.194+0000 7fd89affd640 1 -- 172.21.15.77:0/3814442538 learned_addr learned my addr 172.21.15.77:0/3814442538 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.194+0000 7fd8a3fff640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a406b830 0x7fd8a4070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.194+0000 7fd8a37fe640 1 --2- 172.21.15.77:0/3814442538 >> v2:172.21.15.121:3300/0 conn(0x7fd8a4145730 0x7fd8a415f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.194+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2145295062 0 0) 0x7fd8a415fee0 con 0x7fd8a40b3ac0 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd890003520 con 0x7fd8a40b3ac0 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3684810478 0 0) 0x7fd890003520 con 0x7fd8a40b3ac0 2024-04-25T01:21:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a406b830 msgr2=0x7fd8a4070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a406b830 0x7fd8a4070180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 >> v2:172.21.15.121:3300/0 conn(0x7fd8a4145730 msgr2=0x7fd8a415f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 --2- 172.21.15.77:0/3814442538 >> v2:172.21.15.121:3300/0 conn(0x7fd8a4145730 0x7fd8a415f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a41601c0 con 0x7fd8a40b3ac0 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fd88c001670 con 0x7fd8a40b3ac0 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 >> v1:172.21.15.77:6789/0 conn(0x7fd8a40b3ac0 legacy=0x7fd8a406b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 0x7fd890003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd890004800 0x7fd890004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a406b830 0x7fd8900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a40706c0 con 0x7fd8900039b0 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a4145b10 con 0x7fd890004800 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd89000b9c0 con 0x7fd8a406b830 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a37fe640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 0x7fd890003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd89affd640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd890004800 0x7fd890004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a3fff640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a406b830 0x7fd8900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a37fe640 1 -- 172.21.15.77:0/3814442538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a406b830 msgr2=0x7fd8900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a37fe640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a406b830 0x7fd8900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a37fe640 1 -- 172.21.15.77:0/3814442538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd890004800 msgr2=0x7fd890004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a37fe640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd890004800 0x7fd890004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.195+0000 7fd8a37fe640 1 -- 172.21.15.77:0/3814442538 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd890003520 con 0x7fd8900039b0 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.196+0000 7fd8a37fe640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 0x7fd890003dd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd894009ac0 tx=0x7fd890005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907c80dc79f074fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.196+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd894036070 con 0x7fd8900039b0 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.196+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd894031400 con 0x7fd8900039b0 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.196+0000 7fd8a2ffd640 1 -- 172.21.15.77:0/3814442538 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd894009690 con 0x7fd8900039b0 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.196+0000 7fd8a9e38640 1 -- 172.21.15.77:0/3814442538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 msgr2=0x7fd890003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.196+0000 7fd8a9e38640 1 --2- 172.21.15.77:0/3814442538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 0x7fd890003dd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd894009ac0 tx=0x7fd890005830 comp rx=0 tx=0).stop 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.196+0000 7fd8a3fff640 1 -- 172.21.15.77:0/3814442538 reap_dead start 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.196+0000 7fd8a9e38640 1 -- 172.21.15.77:0/3814442538 shutdown_connections 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.196+0000 7fd8a9e38640 1 -- 172.21.15.77:0/3814442538 >> 172.21.15.77:0/3814442538 conn(0x7fd8a40d0b30 msgr2=0x7fd8a40d0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.196+0000 7fd8a9e38640 1 -- 172.21.15.77:0/3814442538 shutdown_connections 2024-04-25T01:21:47.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.197+0000 7fd8a9e38640 1 -- 172.21.15.77:0/3814442538 wait complete. 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.198+0000 7fd8a9e38640 1 Processor -- start 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.198+0000 7fd8a9e38640 1 -- start start 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.198+0000 7fd8a9e38640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 0x7fd8a40c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.198+0000 7fd8a9e38640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a40c6780 0x7fd8a40c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a9e38640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8a4135dd0 0x7fd8a40c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a9e38640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a4160780 con 0x7fd8900039b0 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a9e38640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a40cb6d0 con 0x7fd8a4135dd0 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a9e38640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a4070180 con 0x7fd8a40c6780 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 0x7fd8a40c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 0x7fd8a40c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40642/0 (socket says 172.21.15.77:40642) 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a37fe640 1 -- 172.21.15.77:0/1879236525 learned_addr learned my addr 172.21.15.77:0/1879236525 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a3fff640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8a4135dd0 0x7fd8a40c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd89affd640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a40c6780 0x7fd8a40c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a37fe640 1 -- 172.21.15.77:0/1879236525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a40c6780 msgr2=0x7fd8a40c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a37fe640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a40c6780 0x7fd8a40c95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a37fe640 1 -- 172.21.15.77:0/1879236525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8a4135dd0 msgr2=0x7fd8a40c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a37fe640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8a4135dd0 0x7fd8a40c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a37fe640 1 -- 172.21.15.77:0/1879236525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a40c6c40 con 0x7fd8900039b0 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd89affd640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a40c6780 0x7fd8a40c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a37fe640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 0x7fd8a40c90b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd894009ac0 tx=0x7fd894004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a0ff9640 1 -- 172.21.15.77:0/1879236525 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd894040020 con 0x7fd8900039b0 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a0ff9640 1 -- 172.21.15.77:0/1879236525 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd894003b40 con 0x7fd8900039b0 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a0ff9640 1 -- 172.21.15.77:0/1879236525 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd894036040 con 0x7fd8900039b0 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a9e38640 1 -- 172.21.15.77:0/1879236525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a40c6f20 con 0x7fd8900039b0 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.199+0000 7fd8a9e38640 1 -- 172.21.15.77:0/1879236525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a40c74e0 con 0x7fd8900039b0 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.200+0000 7fd8a9e38640 1 -- 172.21.15.77:0/1879236525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8a4068fb0 con 0x7fd8900039b0 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.200+0000 7fd8a0ff9640 1 -- 172.21.15.77:0/1879236525 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd894043020 con 0x7fd8900039b0 2024-04-25T01:21:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.201+0000 7fd8a0ff9640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd88406be60 0x7fd88406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:47.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.201+0000 7fd8a0ff9640 1 -- 172.21.15.77:0/1879236525 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7fd8940ab0c0 con 0x7fd8900039b0 2024-04-25T01:21:47.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.201+0000 7fd89affd640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd88406be60 0x7fd88406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:47.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.201+0000 7fd89affd640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd88406be60 0x7fd88406e320 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd8a40c6710 tx=0x7fd88c000f50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:47.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.205+0000 7fd8a0ff9640 1 -- 172.21.15.77:0/1879236525 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd89404a070 con 0x7fd8900039b0 2024-04-25T01:21:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:47.379+0000 7fd8a9e38640 1 -- 172.21.15.77:0/1879236525 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd8a40cbf50 con 0x7fd8900039b0 2024-04-25T01:21:49.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.099+0000 7fd8a0ff9640 1 -- 172.21.15.77:0/1879236525 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7fd8940044e0 con 0x7fd8900039b0 2024-04-25T01:21:49.082 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data_ec' created 2024-04-25T01:21:49.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 -- 172.21.15.77:0/1879236525 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd88406be60 msgr2=0x7fd88406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:49.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd88406be60 0x7fd88406e320 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd8a40c6710 tx=0x7fd88c000f50 comp rx=0 tx=0).stop 2024-04-25T01:21:49.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 -- 172.21.15.77:0/1879236525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 msgr2=0x7fd8a40c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:49.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 0x7fd8a40c90b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd894009ac0 tx=0x7fd894004730 comp rx=0 tx=0).stop 2024-04-25T01:21:49.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 -- 172.21.15.77:0/1879236525 shutdown_connections 2024-04-25T01:21:49.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd88406be60 0x7fd88406e320 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:49.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8a4135dd0 0x7fd8a40c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:49.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a40c6780 0x7fd8a40c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:49.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 --2- 172.21.15.77:0/1879236525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8900039b0 0x7fd8a40c90b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:49.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 -- 172.21.15.77:0/1879236525 >> 172.21.15.77:0/1879236525 conn(0x7fd8a40d0b30 msgr2=0x7fd8a406dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:49.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 -- 172.21.15.77:0/1879236525 shutdown_connections 2024-04-25T01:21:49.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.103+0000 7fd8a9e38640 1 -- 172.21.15.77:0/1879236525 wait complete. 2024-04-25T01:21:49.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-25T01:21:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.228+0000 7f4745d7e640 1 Processor -- start 2024-04-25T01:21:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.228+0000 7f4745d7e640 1 -- start start 2024-04-25T01:21:49.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.229+0000 7f4745d7e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f474006ea00 0x7f4740165b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:49.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.229+0000 7f4745d7e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4740072480 con 0x7f47400ce470 2024-04-25T01:21:49.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.229+0000 7f4745d7e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47400cf700 con 0x7f474015ae90 2024-04-25T01:21:49.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.229+0000 7f4745d7e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f47400cd6b0 con 0x7f474006ea00 2024-04-25T01:21:49.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.229+0000 7f473ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f474006ea00 0x7f4740165b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:49.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.229+0000 7f473effd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f47400ce470 0x7f474006e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53830/0 (socket says 172.21.15.77:53830) 2024-04-25T01:21:49.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.229+0000 7f473effd640 1 -- 172.21.15.77:0/660058575 learned_addr learned my addr 172.21.15.77:0/660058575 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:49.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.229+0000 7f473ffff640 1 -- 172.21.15.77:0/660058575 >> v1:172.21.15.121:6789/0 conn(0x7f474015ae90 legacy=0x7f47400ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:49.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.229+0000 7f473ffff640 1 -- 172.21.15.77:0/660058575 >> v1:172.21.15.77:6789/0 conn(0x7f47400ce470 legacy=0x7f474006e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:49.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.229+0000 7f473ffff640 1 -- 172.21.15.77:0/660058575 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47400cd990 con 0x7f474006ea00 2024-04-25T01:21:49.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.229+0000 7f473e7fc640 1 -- 172.21.15.77:0/660058575 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3463947504 0 0) 0x7f4740072480 con 0x7f47400ce470 2024-04-25T01:21:49.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.230+0000 7f473ffff640 1 --2- 172.21.15.77:0/660058575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f474006ea00 0x7f4740165b90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4738009d90 tx=0x7f473802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e332d97331fc6f6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.230+0000 7f473e7fc640 1 -- 172.21.15.77:0/660058575 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738007740 con 0x7f474006ea00 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.230+0000 7f473e7fc640 1 -- 172.21.15.77:0/660058575 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47380078e0 con 0x7f474006ea00 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.230+0000 7f473e7fc640 1 -- 172.21.15.77:0/660058575 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738007cb0 con 0x7f474006ea00 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.230+0000 7f4745d7e640 1 -- 172.21.15.77:0/660058575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f474006ea00 msgr2=0x7f4740165b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.230+0000 7f4745d7e640 1 --2- 172.21.15.77:0/660058575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f474006ea00 0x7f4740165b90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4738009d90 tx=0x7f473802a6a0 comp rx=0 tx=0).stop 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.231+0000 7f4745d7e640 1 -- 172.21.15.77:0/660058575 shutdown_connections 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.231+0000 7f4745d7e640 1 --2- 172.21.15.77:0/660058575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f474006ea00 0x7f4740165b90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.231+0000 7f4745d7e640 1 -- 172.21.15.77:0/660058575 >> 172.21.15.77:0/660058575 conn(0x7f47400cfae0 msgr2=0x7f47400cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.231+0000 7f4745d7e640 1 -- 172.21.15.77:0/660058575 shutdown_connections 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.231+0000 7f4745d7e640 1 -- 172.21.15.77:0/660058575 wait complete. 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.232+0000 7f4745d7e640 1 Processor -- start 2024-04-25T01:21:49.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.232+0000 7f4745d7e640 1 -- start start 2024-04-25T01:21:49.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f4745d7e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474006ea00 0x7f4740143420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:49.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f4745d7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47400ce470 0x7f474015db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:49.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f4745d7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f474015ae90 0x7f47400b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:49.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f4745d7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f474006cf40 con 0x7f474015ae90 2024-04-25T01:21:49.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f4745d7e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f47400cdf50 con 0x7f474006ea00 2024-04-25T01:21:49.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f4745d7e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f47400cf2a0 con 0x7f47400ce470 2024-04-25T01:21:49.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f474015ae90 0x7f47400b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:49.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f474015ae90 0x7f47400b0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40658/0 (socket says 172.21.15.77:40658) 2024-04-25T01:21:49.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473ffff640 1 -- 172.21.15.77:0/3692080301 learned_addr learned my addr 172.21.15.77:0/3692080301 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:49.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473effd640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47400ce470 0x7f474015db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473f7fe640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474006ea00 0x7f4740143420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473ffff640 1 -- 172.21.15.77:0/3692080301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47400ce470 msgr2=0x7f474015db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473ffff640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47400ce470 0x7f474015db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473ffff640 1 -- 172.21.15.77:0/3692080301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474006ea00 msgr2=0x7f4740143420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473ffff640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474006ea00 0x7f4740143420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473ffff640 1 -- 172.21.15.77:0/3692080301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4740143960 con 0x7f474015ae90 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473f7fe640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474006ea00 0x7f4740143420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473effd640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47400ce470 0x7f474015db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f473ffff640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f474015ae90 0x7f47400b0a10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4738009eb0 tx=0x7f4738004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f4727fff640 1 -- 172.21.15.77:0/3692080301 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738040070 con 0x7f474015ae90 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f4727fff640 1 -- 172.21.15.77:0/3692080301 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4738031c90 con 0x7f474015ae90 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f4727fff640 1 -- 172.21.15.77:0/3692080301 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738045680 con 0x7f474015ae90 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f4745d7e640 1 -- 172.21.15.77:0/3692080301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47400b4ab0 con 0x7f474015ae90 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.233+0000 7f4745d7e640 1 -- 172.21.15.77:0/3692080301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47400b5070 con 0x7f474015ae90 2024-04-25T01:21:49.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.234+0000 7f4745d7e640 1 -- 172.21.15.77:0/3692080301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4740068fb0 con 0x7f474015ae90 2024-04-25T01:21:49.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.235+0000 7f4727fff640 1 -- 172.21.15.77:0/3692080301 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4738039040 con 0x7f474015ae90 2024-04-25T01:21:49.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.235+0000 7f4727fff640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f472c06be30 0x7f472c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:49.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.236+0000 7f4727fff640 1 -- 172.21.15.77:0/3692080301 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7f47380acc00 con 0x7f474015ae90 2024-04-25T01:21:49.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.236+0000 7f473f7fe640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f472c06be30 0x7f472c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:49.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.236+0000 7f473f7fe640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f472c06be30 0x7f472c06e2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4730006fd0 tx=0x7f4730008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:49.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.240+0000 7f4727fff640 1 -- 172.21.15.77:0/3692080301 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f47380b3080 con 0x7f474015ae90 2024-04-25T01:21:49.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:49.415+0000 7f4745d7e640 1 -- 172.21.15.77:0/3692080301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f474006ca90 con 0x7f474015ae90 2024-04-25T01:21:49.805 INFO:tasks.ceph.mds.d.smithi121.stderr:2024-04-25T01:21:49.823+0000 7f279eae4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:21:50.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.101+0000 7f4727fff640 1 -- 172.21.15.77:0/3692080301 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7f473807b3e0 con 0x7f474015ae90 2024-04-25T01:21:50.083 INFO:teuthology.orchestra.run.smithi077.stderr:set pool 4 allow_ec_overwrites to true 2024-04-25T01:21:50.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.105+0000 7f4745d7e640 1 -- 172.21.15.77:0/3692080301 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f472c06be30 msgr2=0x7f472c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:50.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.105+0000 7f4745d7e640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f472c06be30 0x7f472c06e2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4730006fd0 tx=0x7f4730008040 comp rx=0 tx=0).stop 2024-04-25T01:21:50.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.105+0000 7f4745d7e640 1 -- 172.21.15.77:0/3692080301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f474015ae90 msgr2=0x7f47400b0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.105+0000 7f4745d7e640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f474015ae90 0x7f47400b0a10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4738009eb0 tx=0x7f4738004a40 comp rx=0 tx=0).stop 2024-04-25T01:21:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.106+0000 7f4745d7e640 1 -- 172.21.15.77:0/3692080301 shutdown_connections 2024-04-25T01:21:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.106+0000 7f4745d7e640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f472c06be30 0x7f472c06e2f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.106+0000 7f4745d7e640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f474015ae90 0x7f47400b0a10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.106+0000 7f4745d7e640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47400ce470 0x7f474015db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.106+0000 7f4745d7e640 1 --2- 172.21.15.77:0/3692080301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474006ea00 0x7f4740143420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.106+0000 7f4745d7e640 1 -- 172.21.15.77:0/3692080301 >> 172.21.15.77:0/3692080301 conn(0x7f47400cfae0 msgr2=0x7f4740072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.106+0000 7f4745d7e640 1 -- 172.21.15.77:0/3692080301 shutdown_connections 2024-04-25T01:21:50.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.108+0000 7f4745d7e640 1 -- 172.21.15.77:0/3692080301 wait complete. 2024-04-25T01:21:50.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-25T01:21:50.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.241+0000 7fbeebbf9640 1 Processor -- start 2024-04-25T01:21:50.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.242+0000 7fbeebbf9640 1 -- start start 2024-04-25T01:21:50.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.242+0000 7fbeebbf9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee4140d30 0x7fbee415f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:50.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.242+0000 7fbeebbf9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbee4162360 0x7fbee40c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:50.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.242+0000 7fbeebbf9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbee40ca790 con 0x7fbee406b120 2024-04-25T01:21:50.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.242+0000 7fbeebbf9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fbee40caa70 con 0x7fbee4162360 2024-04-25T01:21:50.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.242+0000 7fbeebbf9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbee40cad50 con 0x7fbee4140d30 2024-04-25T01:21:50.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.243+0000 7fbeea196640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fbee406b120 0x7fbee4143610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:52556/0 (socket says 172.21.15.77:52556) 2024-04-25T01:21:50.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.243+0000 7fbee9194640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbee4162360 0x7fbee40c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:50.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.243+0000 7fbee9995640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee4140d30 0x7fbee415f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:50.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.243+0000 7fbeea196640 1 -- 172.21.15.77:0/1865693973 learned_addr learned my addr 172.21.15.77:0/1865693973 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:50.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.243+0000 7fbee9194640 1 -- 172.21.15.77:0/1865693973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee4140d30 msgr2=0x7fbee415f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:50.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.243+0000 7fbee9194640 1 --2- 172.21.15.77:0/1865693973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee4140d30 0x7fbee415f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:50.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.243+0000 7fbee9194640 1 -- 172.21.15.77:0/1865693973 >> v1:172.21.15.121:6789/0 conn(0x7fbee406b120 legacy=0x7fbee4143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:50.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.243+0000 7fbee9194640 1 -- 172.21.15.77:0/1865693973 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee4161b60 con 0x7fbee4162360 2024-04-25T01:21:50.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.243+0000 7fbee9995640 1 --2- 172.21.15.77:0/1865693973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee4140d30 0x7fbee415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:50.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.244+0000 7fbee9194640 1 --2- 172.21.15.77:0/1865693973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee4162360 0x7fbee40c9f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbed40097d0 tx=0x7fbed402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b4f16e63d96aaec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:50.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.244+0000 7fbee8993640 1 -- 172.21.15.77:0/1865693973 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed4037070 con 0x7fbee4162360 2024-04-25T01:21:50.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.244+0000 7fbee8993640 1 -- 172.21.15.77:0/1865693973 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed4032440 con 0x7fbee4162360 2024-04-25T01:21:50.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.244+0000 7fbee8993640 1 -- 172.21.15.77:0/1865693973 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed4032810 con 0x7fbee4162360 2024-04-25T01:21:50.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.244+0000 7fbeebbf9640 1 -- 172.21.15.77:0/1865693973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee4162360 msgr2=0x7fbee40c9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:50.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.244+0000 7fbeebbf9640 1 --2- 172.21.15.77:0/1865693973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee4162360 0x7fbee40c9f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbed40097d0 tx=0x7fbed402a1c0 comp rx=0 tx=0).stop 2024-04-25T01:21:50.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.244+0000 7fbeebbf9640 1 -- 172.21.15.77:0/1865693973 shutdown_connections 2024-04-25T01:21:50.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.244+0000 7fbeebbf9640 1 --2- 172.21.15.77:0/1865693973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee4162360 0x7fbee40c9f40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:50.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.244+0000 7fbeebbf9640 1 --2- 172.21.15.77:0/1865693973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee4140d30 0x7fbee415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:50.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.244+0000 7fbeebbf9640 1 -- 172.21.15.77:0/1865693973 >> 172.21.15.77:0/1865693973 conn(0x7fbee40d0bf0 msgr2=0x7fbee40d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:50.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.245+0000 7fbeebbf9640 1 -- 172.21.15.77:0/1865693973 shutdown_connections 2024-04-25T01:21:50.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.245+0000 7fbeebbf9640 1 -- 172.21.15.77:0/1865693973 wait complete. 2024-04-25T01:21:50.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.246+0000 7fbeebbf9640 1 Processor -- start 2024-04-25T01:21:50.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.246+0000 7fbeebbf9640 1 -- start start 2024-04-25T01:21:50.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.246+0000 7fbeebbf9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee406b120 0x7fbee40b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:50.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.246+0000 7fbeebbf9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee4140d30 0x7fbee4140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:50.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.246+0000 7fbeebbf9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee4162360 0x7fbee41600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:50.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.246+0000 7fbeebbf9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbee41627c0 con 0x7fbee406b120 2024-04-25T01:21:50.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.246+0000 7fbeebbf9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbee40ca480 con 0x7fbee4162360 2024-04-25T01:21:50.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.246+0000 7fbeebbf9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbee4162940 con 0x7fbee4140d30 2024-04-25T01:21:50.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.246+0000 7fbee9995640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee406b120 0x7fbee40b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:50.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbee9995640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee406b120 0x7fbee40b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40664/0 (socket says 172.21.15.77:40664) 2024-04-25T01:21:50.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbee9995640 1 -- 172.21.15.77:0/685435245 learned_addr learned my addr 172.21.15.77:0/685435245 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbeea196640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee4162360 0x7fbee41600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbee9194640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee4140d30 0x7fbee4140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbee9995640 1 -- 172.21.15.77:0/685435245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee4140d30 msgr2=0x7fbee4140610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbee9995640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee4140d30 0x7fbee4140610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbee9995640 1 -- 172.21.15.77:0/685435245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee4162360 msgr2=0x7fbee41600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbee9995640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee4162360 0x7fbee41600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbee9995640 1 -- 172.21.15.77:0/685435245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee4140b50 con 0x7fbee406b120 2024-04-25T01:21:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbee9995640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee406b120 0x7fbee40b4ab0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbee000ebe0 tx=0x7fbee000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbeda7fc640 1 -- 172.21.15.77:0/685435245 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee0017070 con 0x7fbee406b120 2024-04-25T01:21:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbeda7fc640 1 -- 172.21.15.77:0/685435245 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee000cc30 con 0x7fbee406b120 2024-04-25T01:21:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbeebbf9640 1 -- 172.21.15.77:0/685435245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee40b3a10 con 0x7fbee406b120 2024-04-25T01:21:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbeda7fc640 1 -- 172.21.15.77:0/685435245 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee0013470 con 0x7fbee406b120 2024-04-25T01:21:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.247+0000 7fbeebbf9640 1 -- 172.21.15.77:0/685435245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee40b3fd0 con 0x7fbee406b120 2024-04-25T01:21:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.248+0000 7fbeebbf9640 1 -- 172.21.15.77:0/685435245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee4068fb0 con 0x7fbee406b120 2024-04-25T01:21:50.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.249+0000 7fbeda7fc640 1 -- 172.21.15.77:0/685435245 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbee001a070 con 0x7fbee406b120 2024-04-25T01:21:50.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.249+0000 7fbeda7fc640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec806be30 0x7fbec806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:50.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.249+0000 7fbee9194640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec806be30 0x7fbec806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:50.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.249+0000 7fbeda7fc640 1 -- 172.21.15.77:0/685435245 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7fbee008afd0 con 0x7fbee406b120 2024-04-25T01:21:50.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.249+0000 7fbee9194640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec806be30 0x7fbec806e2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbed4034000 tx=0x7fbed402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:50.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.253+0000 7fbeda7fc640 1 -- 172.21.15.77:0/685435245 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbee4140b50 con 0x7fbee406b120 2024-04-25T01:21:50.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:50.463+0000 7fbeebbf9640 1 -- 172.21.15.77:0/685435245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fbee415de40 con 0x7fbee406b120 2024-04-25T01:21:51.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.113+0000 7fbeda7fc640 1 -- 172.21.15.77:0/685435245 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7fbee00590a0 con 0x7fbee406b120 2024-04-25T01:21:51.096 INFO:teuthology.orchestra.run.smithi077.stderr:added data pool 4 to fsmap 2024-04-25T01:21:51.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 -- 172.21.15.77:0/685435245 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec806be30 msgr2=0x7fbec806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec806be30 0x7fbec806e2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbed4034000 tx=0x7fbed402e000 comp rx=0 tx=0).stop 2024-04-25T01:21:51.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 -- 172.21.15.77:0/685435245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee406b120 msgr2=0x7fbee40b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee406b120 0x7fbee40b4ab0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbee000ebe0 tx=0x7fbee000c520 comp rx=0 tx=0).stop 2024-04-25T01:21:51.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 -- 172.21.15.77:0/685435245 shutdown_connections 2024-04-25T01:21:51.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbec806be30 0x7fbec806e2f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbee4162360 0x7fbee41600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbee4140d30 0x7fbee4140610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 --2- 172.21.15.77:0/685435245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee406b120 0x7fbee40b4ab0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 -- 172.21.15.77:0/685435245 >> 172.21.15.77:0/685435245 conn(0x7fbee40d0bf0 msgr2=0x7fbee40b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:51.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 -- 172.21.15.77:0/685435245 shutdown_connections 2024-04-25T01:21:51.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.116+0000 7fbeebbf9640 1 -- 172.21.15.77:0/685435245 wait complete. 2024-04-25T01:21:51.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:21:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.227+0000 7f77839b7640 1 Processor -- start 2024-04-25T01:21:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.228+0000 7f77839b7640 1 -- start start 2024-04-25T01:21:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f77839b7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c0b3040 0x7f777c15f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f77839b7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f777c0aee10 0x7f777c167c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f77839b7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f777c15f5c0 con 0x7f777c0d0b10 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f77839b7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f777c06e480 con 0x7f777c0aee10 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f77839b7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f777c06e720 con 0x7f777c0b3040 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f7780f52640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f777c0d0b10 0x7f777c0b3420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53832/0 (socket says 172.21.15.77:53832) 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f7780f52640 1 -- 172.21.15.77:0/2637143526 learned_addr learned my addr 172.21.15.77:0/2637143526 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4258146760 0 0) 0x7f7778001110 con 0x7f777c0d0b10 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7768003520 con 0x7f777c0d0b10 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f7781f54640 1 --2- 172.21.15.77:0/2637143526 >> v2:172.21.15.121:3300/0 conn(0x7f777c0aee10 0x7f777c167c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f7781753640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c0b3040 0x7f777c15f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1445544290 0 0) 0x7f7768003520 con 0x7f777c0d0b10 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c0b3040 msgr2=0x7f777c15f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c0b3040 0x7f777c15f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 >> v2:172.21.15.121:3300/0 conn(0x7f777c0aee10 msgr2=0x7f777c167c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 --2- 172.21.15.77:0/2637143526 >> v2:172.21.15.121:3300/0 conn(0x7f777c0aee10 0x7f777c167c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c06ea00 con 0x7f777c0d0b10 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f7778001840 con 0x7f777c0d0b10 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 >> v1:172.21.15.77:6789/0 conn(0x7f777c0d0b10 legacy=0x7f777c0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77680039b0 0x7f7768003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 0x7f7768004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c0b3040 0x7f77680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f777c06e480 con 0x7f7768004800 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f777c06e720 con 0x7f77680039b0 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f776800b9c0 con 0x7f777c0b3040 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f7780f52640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 0x7f7768004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.229+0000 7f7781753640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77680039b0 0x7f7768003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.230+0000 7f7781f54640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c0b3040 0x7f77680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.230+0000 7f7780f52640 1 -- 172.21.15.77:0/2637143526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c0b3040 msgr2=0x7f77680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.230+0000 7f7780f52640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c0b3040 0x7f77680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.230+0000 7f7780f52640 1 -- 172.21.15.77:0/2637143526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77680039b0 msgr2=0x7f7768003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.230+0000 7f7780f52640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77680039b0 0x7f7768003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.230+0000 7f7780f52640 1 -- 172.21.15.77:0/2637143526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7768003520 con 0x7f7768004800 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.230+0000 7f7780f52640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 0x7f7768004c80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f777800c490 tx=0x7f7768005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe903c2764906b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.230+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778032020 con 0x7f7768004800 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.230+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7778005ad0 con 0x7f7768004800 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.230+0000 7f776ffff640 1 -- 172.21.15.77:0/2637143526 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777800c0e0 con 0x7f7768004800 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.231+0000 7f77839b7640 1 -- 172.21.15.77:0/2637143526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 msgr2=0x7f7768004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.231+0000 7f77839b7640 1 --2- 172.21.15.77:0/2637143526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 0x7f7768004c80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f777800c490 tx=0x7f7768005830 comp rx=0 tx=0).stop 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.231+0000 7f7781f54640 1 -- 172.21.15.77:0/2637143526 reap_dead start 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.231+0000 7f77839b7640 1 -- 172.21.15.77:0/2637143526 shutdown_connections 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.231+0000 7f77839b7640 1 -- 172.21.15.77:0/2637143526 >> 172.21.15.77:0/2637143526 conn(0x7f777c0cff90 msgr2=0x7f777c0d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.231+0000 7f77839b7640 1 -- 172.21.15.77:0/2637143526 shutdown_connections 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.231+0000 7f77839b7640 1 -- 172.21.15.77:0/2637143526 wait complete. 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.232+0000 7f77839b7640 1 Processor -- start 2024-04-25T01:21:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.232+0000 7f77839b7640 1 -- start start 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f77839b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 0x7f777c142340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f77839b7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c151c50 0x7f777c15b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f77839b7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f777c158820 0x7f777c158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f77839b7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f777c0cfc60 con 0x7f7768004800 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f77839b7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f777c06efc0 con 0x7f777c158820 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f77839b7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f777c0cfde0 con 0x7f777c151c50 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f7781753640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 0x7f777c142340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f7781753640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 0x7f777c142340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40684/0 (socket says 172.21.15.77:40684) 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f7781753640 1 -- 172.21.15.77:0/2322981950 learned_addr learned my addr 172.21.15.77:0/2322981950 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f7781f54640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f777c158820 0x7f777c158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f7780f52640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c151c50 0x7f777c15b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f7781753640 1 -- 172.21.15.77:0/2322981950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c151c50 msgr2=0x7f777c15b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f7781753640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c151c50 0x7f777c15b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f7781753640 1 -- 172.21.15.77:0/2322981950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f777c158820 msgr2=0x7f777c158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f7781753640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f777c158820 0x7f777c158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f7781753640 1 -- 172.21.15.77:0/2322981950 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c142880 con 0x7f7768004800 2024-04-25T01:21:51.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f7781753640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 0x7f777c142340 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7770011960 tx=0x7f7770011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:51.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f776dffb640 1 -- 172.21.15.77:0/2322981950 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777000fa90 con 0x7f7768004800 2024-04-25T01:21:51.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.233+0000 7f776dffb640 1 -- 172.21.15.77:0/2322981950 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f777000fc30 con 0x7f7768004800 2024-04-25T01:21:51.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.234+0000 7f77839b7640 1 -- 172.21.15.77:0/2322981950 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777c15bbd0 con 0x7f7768004800 2024-04-25T01:21:51.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.234+0000 7f776dffb640 1 -- 172.21.15.77:0/2322981950 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7770014440 con 0x7f7768004800 2024-04-25T01:21:51.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.234+0000 7f77839b7640 1 -- 172.21.15.77:0/2322981950 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f777c155f20 con 0x7f7768004800 2024-04-25T01:21:51.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.235+0000 7f77839b7640 1 -- 172.21.15.77:0/2322981950 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f777c068fb0 con 0x7f7768004800 2024-04-25T01:21:51.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.235+0000 7f776dffb640 1 -- 172.21.15.77:0/2322981950 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7770012040 con 0x7f7768004800 2024-04-25T01:21:51.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.235+0000 7f776dffb640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f776006be60 0x7f776006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.236+0000 7f776dffb640 1 -- 172.21.15.77:0/2322981950 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f777008b7d0 con 0x7f7768004800 2024-04-25T01:21:51.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.236+0000 7f7780f52640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f776006be60 0x7f776006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.236+0000 7f7780f52640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f776006be60 0x7f776006e320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7778001050 tx=0x7f7778009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:51.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.240+0000 7f776dffb640 1 -- 172.21.15.77:0/2322981950 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7770019030 con 0x7f7768004800 2024-04-25T01:21:51.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.451+0000 7f77839b7640 1 -- 172.21.15.77:0/2322981950 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f777c15eca0 con 0x7f7768004800 2024-04-25T01:21:51.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.452+0000 7f776dffb640 1 -- 172.21.15.77:0/2322981950 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5180 (secure 0 0 0) 0x7f7770059e40 con 0x7f7768004800 2024-04-25T01:21:51.434 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:51.434 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4267907621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4267907621},{"type":"v1","addr":"172.21.15.77:6837","nonce":4267907621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/493518847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":493518847},{"type":"v1","addr":"172.21.15.121:6833","nonce":493518847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:21:45.408154+0000","modified":"2024-04-25T01:21:50.465658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6835/2810316639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2810316639},{"type":"v1","addr":"172.21.15.121:6835","nonce":2810316639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T01:21:51.434 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 7 2024-04-25T01:21:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 -- 172.21.15.77:0/2322981950 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f776006be60 msgr2=0x7f776006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f776006be60 0x7f776006e320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7778001050 tx=0x7f7778009030 comp rx=0 tx=0).stop 2024-04-25T01:21:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 -- 172.21.15.77:0/2322981950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 msgr2=0x7f777c142340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 0x7f777c142340 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7770011960 tx=0x7f7770011e20 comp rx=0 tx=0).stop 2024-04-25T01:21:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 -- 172.21.15.77:0/2322981950 shutdown_connections 2024-04-25T01:21:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f776006be60 0x7f776006e320 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f777c158820 0x7f777c158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777c151c50 0x7f777c15b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 --2- 172.21.15.77:0/2322981950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768004800 0x7f777c142340 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 -- 172.21.15.77:0/2322981950 >> 172.21.15.77:0/2322981950 conn(0x7f777c0cff90 msgr2=0x7f777c06f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 -- 172.21.15.77:0/2322981950 shutdown_connections 2024-04-25T01:21:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.454+0000 7f77839b7640 1 -- 172.21.15.77:0/2322981950 wait complete. 2024-04-25T01:21:51.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:21:51.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.562+0000 7f3a9379e640 1 Processor -- start 2024-04-25T01:21:51.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.562+0000 7f3a9379e640 1 -- start start 2024-04-25T01:21:51.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9379e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3a8c0ce500 0x7f3a8c0ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9379e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a8c06fa50 0x7f3a8c06feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9379e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a8c0cee20 con 0x7f3a8c161e40 2024-04-25T01:21:51.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9379e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f3a8c0d0f40 con 0x7f3a8c0ce500 2024-04-25T01:21:51.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9379e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3a8c0d0330 con 0x7f3a8c06fa50 2024-04-25T01:21:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9153a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3a8c0ce500 0x7f3a8c0ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9153a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3a8c0ce500 0x7f3a8c0ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40690/0 (socket says 172.21.15.77:40690) 2024-04-25T01:21:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9153a640 1 -- 172.21.15.77:0/3660394338 learned_addr learned my addr 172.21.15.77:0/3660394338 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9153a640 1 -- 172.21.15.77:0/3660394338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a8c06fa50 msgr2=0x7f3a8c06feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9153a640 1 --2- 172.21.15.77:0/3660394338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a8c06fa50 0x7f3a8c06feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9153a640 1 -- 172.21.15.77:0/3660394338 >> v1:172.21.15.121:6789/0 conn(0x7f3a8c161e40 legacy=0x7f3a8c1622d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.563+0000 7f3a9153a640 1 -- 172.21.15.77:0/3660394338 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a8c0d0610 con 0x7f3a8c0ce500 2024-04-25T01:21:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.564+0000 7f3a9153a640 1 --2- 172.21.15.77:0/3660394338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a8c0ce500 0x7f3a8c0ce8e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0098a0 tx=0x7f3a7c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6417a80d0c8c95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.564+0000 7f3a8bfff640 1 -- 172.21.15.77:0/3660394338 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c037070 con 0x7f3a8c0ce500 2024-04-25T01:21:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.564+0000 7f3a8bfff640 1 -- 172.21.15.77:0/3660394338 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a7c032440 con 0x7f3a8c0ce500 2024-04-25T01:21:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.564+0000 7f3a8bfff640 1 -- 172.21.15.77:0/3660394338 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c032810 con 0x7f3a8c0ce500 2024-04-25T01:21:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.565+0000 7f3a9379e640 1 -- 172.21.15.77:0/3660394338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a8c0ce500 msgr2=0x7f3a8c0ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.565+0000 7f3a9379e640 1 --2- 172.21.15.77:0/3660394338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a8c0ce500 0x7f3a8c0ce8e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0098a0 tx=0x7f3a7c02a250 comp rx=0 tx=0).stop 2024-04-25T01:21:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.565+0000 7f3a9379e640 1 -- 172.21.15.77:0/3660394338 shutdown_connections 2024-04-25T01:21:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.565+0000 7f3a9379e640 1 --2- 172.21.15.77:0/3660394338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a8c06fa50 0x7f3a8c06feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.565+0000 7f3a9379e640 1 --2- 172.21.15.77:0/3660394338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a8c0ce500 0x7f3a8c0ce8e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.565+0000 7f3a9379e640 1 -- 172.21.15.77:0/3660394338 >> 172.21.15.77:0/3660394338 conn(0x7f3a8c0d1c80 msgr2=0x7f3a8c0d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.565+0000 7f3a9379e640 1 -- 172.21.15.77:0/3660394338 shutdown_connections 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.565+0000 7f3a9379e640 1 -- 172.21.15.77:0/3660394338 wait complete. 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.566+0000 7f3a9379e640 1 Processor -- start 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.566+0000 7f3a9379e640 1 -- start start 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a9379e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a8c06fa50 0x7f3a8c15f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a9379e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a8c0ce500 0x7f3a8c150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a9379e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a8c161e40 0x7f3a8c151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a9379e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8c0b1390 con 0x7f3a8c161e40 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a9379e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8c0d0bd0 con 0x7f3a8c0ce500 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a9379e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3a8c0b1510 con 0x7f3a8c06fa50 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a91d3b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a8c161e40 0x7f3a8c151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a91d3b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a8c161e40 0x7f3a8c151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40694/0 (socket says 172.21.15.77:40694) 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a91d3b640 1 -- 172.21.15.77:0/221070707 learned_addr learned my addr 172.21.15.77:0/221070707 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a9153a640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a8c06fa50 0x7f3a8c15f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a90d39640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a8c0ce500 0x7f3a8c150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a91d3b640 1 -- 172.21.15.77:0/221070707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a8c06fa50 msgr2=0x7f3a8c15f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a91d3b640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a8c06fa50 0x7f3a8c15f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a91d3b640 1 -- 172.21.15.77:0/221070707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a8c0ce500 msgr2=0x7f3a8c150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a91d3b640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a8c0ce500 0x7f3a8c150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a91d3b640 1 -- 172.21.15.77:0/221070707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a8c15fd30 con 0x7f3a8c161e40 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a90d39640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a8c0ce500 0x7f3a8c150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a9153a640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a8c06fa50 0x7f3a8c15f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a91d3b640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a8c161e40 0x7f3a8c151190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400b990 tx=0x7f3a8400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.567+0000 7f3a89ffb640 1 -- 172.21.15.77:0/221070707 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a84019070 con 0x7f3a8c161e40 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.568+0000 7f3a89ffb640 1 -- 172.21.15.77:0/221070707 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a8400cba0 con 0x7f3a8c161e40 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.568+0000 7f3a9379e640 1 -- 172.21.15.77:0/221070707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a8c160010 con 0x7f3a8c161e40 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.568+0000 7f3a89ffb640 1 -- 172.21.15.77:0/221070707 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a84014440 con 0x7f3a8c161e40 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.568+0000 7f3a9379e640 1 -- 172.21.15.77:0/221070707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a8c14d700 con 0x7f3a8c161e40 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.569+0000 7f3a9379e640 1 -- 172.21.15.77:0/221070707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a8c068180 con 0x7f3a8c161e40 2024-04-25T01:21:51.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.569+0000 7f3a89ffb640 1 -- 172.21.15.77:0/221070707 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3a8401c070 con 0x7f3a8c161e40 2024-04-25T01:21:51.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.569+0000 7f3a89ffb640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3a7806be30 0x7f3a7806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.569+0000 7f3a89ffb640 1 -- 172.21.15.77:0/221070707 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3a8408aeb0 con 0x7f3a8c161e40 2024-04-25T01:21:51.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.569+0000 7f3a9153a640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3a7806be30 0x7f3a7806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.570+0000 7f3a9153a640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3a7806be30 0x7f3a7806e2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c034000 tx=0x7f3a7c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:51.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.574+0000 7f3a89ffb640 1 -- 172.21.15.77:0/221070707 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3a840595b0 con 0x7f3a8c161e40 2024-04-25T01:21:51.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.746+0000 7f3a9379e640 1 -- 172.21.15.77:0/221070707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a8c1601e0 con 0x7f3a8c161e40 2024-04-25T01:21:51.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.747+0000 7f3a89ffb640 1 -- 172.21.15.77:0/221070707 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f3a84058eb0 con 0x7f3a8c161e40 2024-04-25T01:21:51.730 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:51.730 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:21:51.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.749+0000 7f3a9379e640 1 -- 172.21.15.77:0/221070707 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3a7806be30 msgr2=0x7f3a7806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.749+0000 7f3a9379e640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3a7806be30 0x7f3a7806e2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c034000 tx=0x7f3a7c02e000 comp rx=0 tx=0).stop 2024-04-25T01:21:51.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.749+0000 7f3a9379e640 1 -- 172.21.15.77:0/221070707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a8c161e40 msgr2=0x7f3a8c151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.749+0000 7f3a9379e640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a8c161e40 0x7f3a8c151190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400b990 tx=0x7f3a8400be50 comp rx=0 tx=0).stop 2024-04-25T01:21:51.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.749+0000 7f3a9379e640 1 -- 172.21.15.77:0/221070707 shutdown_connections 2024-04-25T01:21:51.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.749+0000 7f3a9379e640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3a7806be30 0x7f3a7806e2f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.750+0000 7f3a9379e640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a8c161e40 0x7f3a8c151190 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.750+0000 7f3a9379e640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a8c0ce500 0x7f3a8c150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.750+0000 7f3a9379e640 1 --2- 172.21.15.77:0/221070707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a8c06fa50 0x7f3a8c15f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.750+0000 7f3a9379e640 1 -- 172.21.15.77:0/221070707 >> 172.21.15.77:0/221070707 conn(0x7f3a8c0d1c80 msgr2=0x7f3a8c0aeaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:51.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.750+0000 7f3a9379e640 1 -- 172.21.15.77:0/221070707 shutdown_connections 2024-04-25T01:21:51.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.750+0000 7f3a9379e640 1 -- 172.21.15.77:0/221070707 wait complete. 2024-04-25T01:21:51.743 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:21:51.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.856+0000 7f1b1b831640 1 Processor -- start 2024-04-25T01:21:51.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.857+0000 7f1b1b831640 1 -- start start 2024-04-25T01:21:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.857+0000 7f1b1b831640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1b140ce500 0x7f1b140ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.857+0000 7f1b1b831640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b1406fee0 0x7f1b14162550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.857+0000 7f1b1b831640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b140cee20 con 0x7f1b1406fa50 2024-04-25T01:21:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.857+0000 7f1b1b831640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f1b140d0f40 con 0x7f1b1406fee0 2024-04-25T01:21:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.857+0000 7f1b1b831640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1b140d0330 con 0x7f1b140ce500 2024-04-25T01:21:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.857+0000 7f1b19dce640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b1406fee0 0x7f1b14162550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.857+0000 7f1b18dcc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1b1406fa50 0x7f1b14161e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53848/0 (socket says 172.21.15.77:53848) 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.857+0000 7f1b18dcc640 1 -- 172.21.15.77:0/2832994608 learned_addr learned my addr 172.21.15.77:0/2832994608 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.858+0000 7f1b195cd640 1 --2- 172.21.15.77:0/2832994608 >> v2:172.21.15.121:3300/0 conn(0x7f1b140ce500 0x7f1b140ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.858+0000 7f1b19dce640 1 -- 172.21.15.77:0/2832994608 >> v2:172.21.15.121:3300/0 conn(0x7f1b140ce500 msgr2=0x7f1b140ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.858+0000 7f1b19dce640 1 --2- 172.21.15.77:0/2832994608 >> v2:172.21.15.121:3300/0 conn(0x7f1b140ce500 0x7f1b140ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.858+0000 7f1b19dce640 1 -- 172.21.15.77:0/2832994608 >> v1:172.21.15.77:6789/0 conn(0x7f1b1406fa50 legacy=0x7f1b14161e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.858+0000 7f1b0bfff640 1 -- 172.21.15.77:0/2832994608 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2766354297 0 0) 0x7f1b140cee20 con 0x7f1b1406fa50 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.858+0000 7f1b19dce640 1 -- 172.21.15.77:0/2832994608 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b140d0610 con 0x7f1b1406fee0 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.858+0000 7f1b19dce640 1 --2- 172.21.15.77:0/2832994608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b1406fee0 0x7f1b14162550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1b14071c00 tx=0x7f1b1002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f781f18c8a2e8821 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.859+0000 7f1b0bfff640 1 -- 172.21.15.77:0/2832994608 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b10037070 con 0x7f1b1406fee0 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.859+0000 7f1b0bfff640 1 -- 172.21.15.77:0/2832994608 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b10032440 con 0x7f1b1406fee0 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.859+0000 7f1b0bfff640 1 -- 172.21.15.77:0/2832994608 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b10032810 con 0x7f1b1406fee0 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.859+0000 7f1b1b831640 1 -- 172.21.15.77:0/2832994608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b1406fee0 msgr2=0x7f1b14162550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.859+0000 7f1b1b831640 1 --2- 172.21.15.77:0/2832994608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b1406fee0 0x7f1b14162550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1b14071c00 tx=0x7f1b1002a1a0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.859+0000 7f1b1b831640 1 -- 172.21.15.77:0/2832994608 shutdown_connections 2024-04-25T01:21:51.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.859+0000 7f1b1b831640 1 --2- 172.21.15.77:0/2832994608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b1406fee0 0x7f1b14162550 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.859+0000 7f1b1b831640 1 --2- 172.21.15.77:0/2832994608 >> v2:172.21.15.121:3300/0 conn(0x7f1b140ce500 0x7f1b140ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.859+0000 7f1b1b831640 1 -- 172.21.15.77:0/2832994608 >> 172.21.15.77:0/2832994608 conn(0x7f1b140d1c80 msgr2=0x7f1b140d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:51.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.859+0000 7f1b1b831640 1 -- 172.21.15.77:0/2832994608 shutdown_connections 2024-04-25T01:21:51.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.859+0000 7f1b1b831640 1 -- 172.21.15.77:0/2832994608 wait complete. 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.860+0000 7f1b1b831640 1 Processor -- start 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.860+0000 7f1b1b831640 1 -- start start 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b1b831640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b1406fa50 0x7f1b14157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b1b831640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b1406fee0 0x7f1b140c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b1b831640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b140ce500 0x7f1b140c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b1b831640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1b140b1390 con 0x7f1b140ce500 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b1b831640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1b140d0bd0 con 0x7f1b1406fee0 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b1b831640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1b140b1510 con 0x7f1b1406fa50 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b19dce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b140ce500 0x7f1b140c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b19dce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b140ce500 0x7f1b140c9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40714/0 (socket says 172.21.15.77:40714) 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b19dce640 1 -- 172.21.15.77:0/964198581 learned_addr learned my addr 172.21.15.77:0/964198581 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b18dcc640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b1406fee0 0x7f1b140c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b19dce640 1 -- 172.21.15.77:0/964198581 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b1406fa50 msgr2=0x7f1b14157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b19dce640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b1406fa50 0x7f1b14157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b19dce640 1 -- 172.21.15.77:0/964198581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b1406fee0 msgr2=0x7f1b140c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b19dce640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b1406fee0 0x7f1b140c9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.861+0000 7f1b19dce640 1 -- 172.21.15.77:0/964198581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b140b59d0 con 0x7f1b140ce500 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.862+0000 7f1b19dce640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b140ce500 0x7f1b140c9ba0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009890 tx=0x7f1b100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.862+0000 7f1b09ffb640 1 -- 172.21.15.77:0/964198581 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b10037050 con 0x7f1b140ce500 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.862+0000 7f1b09ffb640 1 -- 172.21.15.77:0/964198581 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b10038040 con 0x7f1b140ce500 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.862+0000 7f1b09ffb640 1 -- 172.21.15.77:0/964198581 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b10004490 con 0x7f1b140ce500 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.862+0000 7f1b1b831640 1 -- 172.21.15.77:0/964198581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b140b5cb0 con 0x7f1b140ce500 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.862+0000 7f1b1b831640 1 -- 172.21.15.77:0/964198581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b140b6270 con 0x7f1b140ce500 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.863+0000 7f1b1b831640 1 -- 172.21.15.77:0/964198581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b140d1430 con 0x7f1b140ce500 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.863+0000 7f1b09ffb640 1 -- 172.21.15.77:0/964198581 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1b10030980 con 0x7f1b140ce500 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.864+0000 7f1b09ffb640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1af806be60 0x7f1af806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.864+0000 7f1b09ffb640 1 -- 172.21.15.77:0/964198581 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1b100abce0 con 0x7f1b140ce500 2024-04-25T01:21:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.864+0000 7f1b195cd640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1af806be60 0x7f1af806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.864+0000 7f1b195cd640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1af806be60 0x7f1af806e320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c007dc0 tx=0x7f1b0c0072f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:51.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:51.868+0000 7f1b09ffb640 1 -- 172.21.15.77:0/964198581 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1b140d1430 con 0x7f1b140ce500 2024-04-25T01:21:52.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.040+0000 7f1b1b831640 1 -- 172.21.15.77:0/964198581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b14068180 con 0x7f1b140ce500 2024-04-25T01:21:52.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.041+0000 7f1b09ffb640 1 -- 172.21.15.77:0/964198581 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f1b10032e20 con 0x7f1b140ce500 2024-04-25T01:21:52.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:52.024 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:21:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.043+0000 7f1b1b831640 1 -- 172.21.15.77:0/964198581 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1af806be60 msgr2=0x7f1af806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.044+0000 7f1b1b831640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1af806be60 0x7f1af806e320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c007dc0 tx=0x7f1b0c0072f0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.044+0000 7f1b1b831640 1 -- 172.21.15.77:0/964198581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b140ce500 msgr2=0x7f1b140c9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.044+0000 7f1b1b831640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b140ce500 0x7f1b140c9ba0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009890 tx=0x7f1b100046c0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.044+0000 7f1b1b831640 1 -- 172.21.15.77:0/964198581 shutdown_connections 2024-04-25T01:21:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.044+0000 7f1b1b831640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1af806be60 0x7f1af806e320 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.044+0000 7f1b1b831640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b140ce500 0x7f1b140c9ba0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.044+0000 7f1b1b831640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b1406fee0 0x7f1b140c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.044+0000 7f1b1b831640 1 --2- 172.21.15.77:0/964198581 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b1406fa50 0x7f1b14157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.044+0000 7f1b1b831640 1 -- 172.21.15.77:0/964198581 >> 172.21.15.77:0/964198581 conn(0x7f1b140d1c80 msgr2=0x7f1b140b2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.044+0000 7f1b1b831640 1 -- 172.21.15.77:0/964198581 shutdown_connections 2024-04-25T01:21:52.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.044+0000 7f1b1b831640 1 -- 172.21.15.77:0/964198581 wait complete. 2024-04-25T01:21:52.050 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:21:52.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.167+0000 7f64e1a12640 1 Processor -- start 2024-04-25T01:21:52.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.167+0000 7f64e1a12640 1 -- start start 2024-04-25T01:21:52.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64e1a12640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0cfaf0 0x7f64dc0d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64e1a12640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64dc0d1e90 con 0x7f64dc15e4c0 2024-04-25T01:21:52.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64e1a12640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64dc0b54a0 con 0x7f64dc0d4a10 2024-04-25T01:21:52.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64e1a12640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64dc0d2130 con 0x7f64dc0cfaf0 2024-04-25T01:21:52.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64daffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f64dc15e4c0 0x7f64dc0d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53862/0 (socket says 172.21.15.77:53862) 2024-04-25T01:21:52.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64daffd640 1 -- 172.21.15.77:0/3658104455 learned_addr learned my addr 172.21.15.77:0/3658104455 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64db7fe640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0cfaf0 0x7f64dc0d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922670218 0 0) 0x7f64dc0d1e90 con 0x7f64dc15e4c0 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c8003520 con 0x7f64dc15e4c0 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 153993508 0 0) 0x7f64c8003520 con 0x7f64dc15e4c0 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0cfaf0 msgr2=0x7f64dc0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0cfaf0 0x7f64dc0d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 >> v1:172.21.15.121:6789/0 conn(0x7f64dc0d4a10 legacy=0x7f64dc0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc0b5780 con 0x7f64dc15e4c0 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f64cc001670 con 0x7f64dc15e4c0 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 >> v1:172.21.15.77:6789/0 conn(0x7f64dc15e4c0 legacy=0x7f64dc0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64c80039e0 0x7f64c8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64c8004800 0x7f64c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0cfaf0 0x7f64c8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc0d2130 con 0x7f64c80039e0 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f64c800baf0 con 0x7f64c8004800 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64c800bcf0 con 0x7f64dc0cfaf0 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.168+0000 7f64daffd640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64c80039e0 0x7f64c8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.169+0000 7f64db7fe640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0cfaf0 0x7f64c8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.169+0000 7f64d3fff640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64c8004800 0x7f64c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.169+0000 7f64daffd640 1 -- 172.21.15.77:0/3658104455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0cfaf0 msgr2=0x7f64c8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.169+0000 7f64daffd640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0cfaf0 0x7f64c8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.169+0000 7f64daffd640 1 -- 172.21.15.77:0/3658104455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64c8004800 msgr2=0x7f64c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.169+0000 7f64daffd640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64c8004800 0x7f64c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.169+0000 7f64daffd640 1 -- 172.21.15.77:0/3658104455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c8003520 con 0x7f64c80039e0 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.169+0000 7f64daffd640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64c80039e0 0x7f64c8003e00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00c210 tx=0x7f64c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77221a0ce36996f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.169+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc007d30 con 0x7f64c80039e0 2024-04-25T01:21:52.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.169+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64cc005ad0 con 0x7f64c80039e0 2024-04-25T01:21:52.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.169+0000 7f64da7fc640 1 -- 172.21.15.77:0/3658104455 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc007070 con 0x7f64c80039e0 2024-04-25T01:21:52.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.170+0000 7f64e1a12640 1 -- 172.21.15.77:0/3658104455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64c80039e0 msgr2=0x7f64c8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.170+0000 7f64e1a12640 1 --2- 172.21.15.77:0/3658104455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64c80039e0 0x7f64c8003e00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00c210 tx=0x7f64c8005830 comp rx=0 tx=0).stop 2024-04-25T01:21:52.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.170+0000 7f64db7fe640 1 -- 172.21.15.77:0/3658104455 reap_dead start 2024-04-25T01:21:52.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.170+0000 7f64e1a12640 1 -- 172.21.15.77:0/3658104455 shutdown_connections 2024-04-25T01:21:52.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.170+0000 7f64e1a12640 1 -- 172.21.15.77:0/3658104455 >> 172.21.15.77:0/3658104455 conn(0x7f64dc0d3ef0 msgr2=0x7f64dc0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:52.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.170+0000 7f64e1a12640 1 -- 172.21.15.77:0/3658104455 shutdown_connections 2024-04-25T01:21:52.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.170+0000 7f64e1a12640 1 -- 172.21.15.77:0/3658104455 wait complete. 2024-04-25T01:21:52.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.171+0000 7f64e1a12640 1 Processor -- start 2024-04-25T01:21:52.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.171+0000 7f64e1a12640 1 -- start start 2024-04-25T01:21:52.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64e1a12640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64c80039e0 0x7f64dc0cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64e1a12640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0b9240 0x7f64dc0b96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64e1a12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64dc0b4350 0x7f64dc0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64e1a12640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc0b30a0 con 0x7f64dc0b4350 2024-04-25T01:21:52.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64e1a12640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc0b2f20 con 0x7f64c80039e0 2024-04-25T01:21:52.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64e1a12640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64dc15e8a0 con 0x7f64dc0b9240 2024-04-25T01:21:52.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64db7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64dc0b4350 0x7f64dc0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64db7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64dc0b4350 0x7f64dc0b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40738/0 (socket says 172.21.15.77:40738) 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64db7fe640 1 -- 172.21.15.77:0/4136328899 learned_addr learned my addr 172.21.15.77:0/4136328899 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64db7fe640 1 -- 172.21.15.77:0/4136328899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0b9240 msgr2=0x7f64dc0b96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64db7fe640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0b9240 0x7f64dc0b96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64db7fe640 1 -- 172.21.15.77:0/4136328899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64c80039e0 msgr2=0x7f64dc0cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64db7fe640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64c80039e0 0x7f64dc0cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64db7fe640 1 -- 172.21.15.77:0/4136328899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc0cdac0 con 0x7f64dc0b4350 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64db7fe640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64dc0b4350 0x7f64dc0b47d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0c79f0 tx=0x7f64d400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.172+0000 7f64d8ff9640 1 -- 172.21.15.77:0/4136328899 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d4015430 con 0x7f64dc0b4350 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.173+0000 7f64d8ff9640 1 -- 172.21.15.77:0/4136328899 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64d401d070 con 0x7f64dc0b4350 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.173+0000 7f64e1a12640 1 -- 172.21.15.77:0/4136328899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64dc0b7f00 con 0x7f64dc0b4350 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.173+0000 7f64d8ff9640 1 -- 172.21.15.77:0/4136328899 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d4015830 con 0x7f64dc0b4350 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.173+0000 7f64e1a12640 1 -- 172.21.15.77:0/4136328899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64dc0b84c0 con 0x7f64dc0b4350 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.174+0000 7f64e1a12640 1 -- 172.21.15.77:0/4136328899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64dc068fb0 con 0x7f64dc0b4350 2024-04-25T01:21:52.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.174+0000 7f64d8ff9640 1 -- 172.21.15.77:0/4136328899 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f64d40159d0 con 0x7f64dc0b4350 2024-04-25T01:21:52.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.174+0000 7f64d8ff9640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64bc06be60 0x7f64bc06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.174+0000 7f64d8ff9640 1 -- 172.21.15.77:0/4136328899 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f64d408bab0 con 0x7f64dc0b4350 2024-04-25T01:21:52.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.174+0000 7f64daffd640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64bc06be60 0x7f64bc06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.175+0000 7f64daffd640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64bc06be60 0x7f64bc06e320 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f64cc033000 tx=0x7f64cc009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:52.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.179+0000 7f64d8ff9640 1 -- 172.21.15.77:0/4136328899 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f64dc0cdac0 con 0x7f64dc0b4350 2024-04-25T01:21:52.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.386+0000 7f64e1a12640 1 -- 172.21.15.77:0/4136328899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64dc06bb10 con 0x7f64dc0b4350 2024-04-25T01:21:52.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.387+0000 7f64d8ff9640 1 -- 172.21.15.77:0/4136328899 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5180 (secure 0 0 0) 0x7f64d4010040 con 0x7f64dc0b4350 2024-04-25T01:21:52.369 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:52.369 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4267907621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4267907621},{"type":"v1","addr":"172.21.15.77:6837","nonce":4267907621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/493518847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":493518847},{"type":"v1","addr":"172.21.15.121:6833","nonce":493518847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:21:45.408154+0000","modified":"2024-04-25T01:21:50.465658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6835/2810316639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2810316639},{"type":"v1","addr":"172.21.15.121:6835","nonce":2810316639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T01:21:52.369 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 7 2024-04-25T01:21:52.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 -- 172.21.15.77:0/4136328899 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64bc06be60 msgr2=0x7f64bc06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64bc06be60 0x7f64bc06e320 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f64cc033000 tx=0x7f64cc009030 comp rx=0 tx=0).stop 2024-04-25T01:21:52.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 -- 172.21.15.77:0/4136328899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64dc0b4350 msgr2=0x7f64dc0b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64dc0b4350 0x7f64dc0b47d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0c79f0 tx=0x7f64d400f580 comp rx=0 tx=0).stop 2024-04-25T01:21:52.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 -- 172.21.15.77:0/4136328899 shutdown_connections 2024-04-25T01:21:52.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64bc06be60 0x7f64bc06e320 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64dc0b4350 0x7f64dc0b47d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64dc0b9240 0x7f64dc0b96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 --2- 172.21.15.77:0/4136328899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64c80039e0 0x7f64dc0cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 -- 172.21.15.77:0/4136328899 >> 172.21.15.77:0/4136328899 conn(0x7f64dc0d3ef0 msgr2=0x7f64dc06c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:52.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 -- 172.21.15.77:0/4136328899 shutdown_connections 2024-04-25T01:21:52.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.389+0000 7f64e1a12640 1 -- 172.21.15.77:0/4136328899 wait complete. 2024-04-25T01:21:52.382 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:21:52.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.496+0000 7f0ef4038640 1 Processor -- start 2024-04-25T01:21:52.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.496+0000 7f0ef4038640 1 -- start start 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef4038640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec0ce500 0x7f0eec0ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef4038640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0eec06fa50 0x7f0eec06feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef4038640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eec0cee20 con 0x7f0eec161e40 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef4038640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0eec0d0f40 con 0x7f0eec06fa50 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef4038640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0eec0d0330 con 0x7f0eec0ce500 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef25d5640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0eec161e40 0x7f0eec1622d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53874/0 (socket says 172.21.15.77:53874) 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef25d5640 1 -- 172.21.15.77:0/3468668385 learned_addr learned my addr 172.21.15.77:0/3468668385 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968570635 0 0) 0x7f0eec0cee20 con 0x7f0eec161e40 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ed8003520 con 0x7f0eec161e40 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef15d3640 1 --2- 172.21.15.77:0/3468668385 >> v2:172.21.15.121:3300/0 conn(0x7f0eec06fa50 0x7f0eec06feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef1dd4640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec0ce500 0x7f0eec0ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1981853623 0 0) 0x7f0ed8003520 con 0x7f0eec161e40 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec0ce500 msgr2=0x7f0eec0ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec0ce500 0x7f0eec0ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 >> v2:172.21.15.121:3300/0 conn(0x7f0eec06fa50 msgr2=0x7f0eec06feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 --2- 172.21.15.77:0/3468668385 >> v2:172.21.15.121:3300/0 conn(0x7f0eec06fa50 0x7f0eec06feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eec0d0610 con 0x7f0eec161e40 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f0ee40019f0 con 0x7f0eec161e40 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 >> v1:172.21.15.77:6789/0 conn(0x7f0eec161e40 legacy=0x7f0eec1622d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 0x7f0ed8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec0ce500 0x7f0ed8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8004d40 0x7f0ed80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec0d0f40 con 0x7f0ed80039b0 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec0d0330 con 0x7f0ed8004d40 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.497+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed800b9c0 con 0x7f0eec0ce500 2024-04-25T01:21:52.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.498+0000 7f0ef1dd4640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 0x7f0ed8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.498+0000 7f0ef25d5640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8004d40 0x7f0ed80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.498+0000 7f0ef15d3640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec0ce500 0x7f0ed8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.498+0000 7f0ef1dd4640 1 -- 172.21.15.77:0/3468668385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec0ce500 msgr2=0x7f0ed8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.498+0000 7f0ef1dd4640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec0ce500 0x7f0ed8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.498+0000 7f0ef1dd4640 1 -- 172.21.15.77:0/3468668385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8004d40 msgr2=0x7f0ed80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.498+0000 7f0ef1dd4640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8004d40 0x7f0ed80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.498+0000 7f0ef1dd4640 1 -- 172.21.15.77:0/3468668385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed8003520 con 0x7f0ed80039b0 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.498+0000 7f0ef1dd4640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 0x7f0ed8003dd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009ac0 tx=0x7f0ed8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fca277b82a5fb3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.499+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8036070 con 0x7f0ed80039b0 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.499+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ee8031400 con 0x7f0ed80039b0 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.499+0000 7f0ef0dd2640 1 -- 172.21.15.77:0/3468668385 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8009690 con 0x7f0ed80039b0 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.499+0000 7f0ef4038640 1 -- 172.21.15.77:0/3468668385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 msgr2=0x7f0ed8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.499+0000 7f0ef4038640 1 --2- 172.21.15.77:0/3468668385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 0x7f0ed8003dd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009ac0 tx=0x7f0ed8005830 comp rx=0 tx=0).stop 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.499+0000 7f0ef25d5640 1 -- 172.21.15.77:0/3468668385 reap_dead start 2024-04-25T01:21:52.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.499+0000 7f0ef4038640 1 -- 172.21.15.77:0/3468668385 shutdown_connections 2024-04-25T01:21:52.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.499+0000 7f0ef4038640 1 -- 172.21.15.77:0/3468668385 >> 172.21.15.77:0/3468668385 conn(0x7f0eec0d1c80 msgr2=0x7f0eec0d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:52.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.499+0000 7f0ef4038640 1 -- 172.21.15.77:0/3468668385 shutdown_connections 2024-04-25T01:21:52.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.499+0000 7f0ef4038640 1 -- 172.21.15.77:0/3468668385 wait complete. 2024-04-25T01:21:52.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.500+0000 7f0ef4038640 1 Processor -- start 2024-04-25T01:21:52.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef4038640 1 -- start start 2024-04-25T01:21:52.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef4038640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 0x7f0eec153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef4038640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec152c40 0x7f0eec06db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef4038640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eec152440 0x7f0eec0aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef4038640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec0b1390 con 0x7f0ed80039b0 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef4038640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec0d0bd0 con 0x7f0eec152440 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef4038640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0eec0b1510 con 0x7f0eec152c40 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef1dd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 0x7f0eec153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef1dd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 0x7f0eec153170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40760/0 (socket says 172.21.15.77:40760) 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef1dd4640 1 -- 172.21.15.77:0/170933982 learned_addr learned my addr 172.21.15.77:0/170933982 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef15d3640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec152c40 0x7f0eec06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.501+0000 7f0ef25d5640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eec152440 0x7f0eec0aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ef1dd4640 1 -- 172.21.15.77:0/170933982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec152c40 msgr2=0x7f0eec06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ef1dd4640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec152c40 0x7f0eec06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ef1dd4640 1 -- 172.21.15.77:0/170933982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eec152440 msgr2=0x7f0eec0aeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ef1dd4640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eec152440 0x7f0eec0aeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ef1dd4640 1 -- 172.21.15.77:0/170933982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eec06e060 con 0x7f0ed80039b0 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ef25d5640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eec152440 0x7f0eec0aeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ef1dd4640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 0x7f0eec153170 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009ac0 tx=0x7f0ee8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ee27fc640 1 -- 172.21.15.77:0/170933982 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8040020 con 0x7f0ed80039b0 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ee27fc640 1 -- 172.21.15.77:0/170933982 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ee8003bc0 con 0x7f0ed80039b0 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ef4038640 1 -- 172.21.15.77:0/170933982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eec15fea0 con 0x7f0ed80039b0 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ee27fc640 1 -- 172.21.15.77:0/170933982 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8036040 con 0x7f0ed80039b0 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.502+0000 7f0ef4038640 1 -- 172.21.15.77:0/170933982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eec160460 con 0x7f0ed80039b0 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.503+0000 7f0ef4038640 1 -- 172.21.15.77:0/170933982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eec0b0340 con 0x7f0ed80039b0 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.503+0000 7f0ee27fc640 1 -- 172.21.15.77:0/170933982 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0ee8043020 con 0x7f0ed80039b0 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.504+0000 7f0ee27fc640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ed006be30 0x7f0ed006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.504+0000 7f0ee27fc640 1 -- 172.21.15.77:0/170933982 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0ee80ab240 con 0x7f0ed80039b0 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.504+0000 7f0ef15d3640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ed006be30 0x7f0ed006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.504+0000 7f0ef15d3640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ed006be30 0x7f0ed006e2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0eec158450 tx=0x7f0edc016040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:52.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.508+0000 7f0ee27fc640 1 -- 172.21.15.77:0/170933982 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0eec0b0340 con 0x7f0ed80039b0 2024-04-25T01:21:52.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.682+0000 7f0ef4038640 1 -- 172.21.15.77:0/170933982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0eec068180 con 0x7f0ed80039b0 2024-04-25T01:21:52.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.683+0000 7f0ee27fc640 1 -- 172.21.15.77:0/170933982 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f0ee80a2e20 con 0x7f0ed80039b0 2024-04-25T01:21:52.665 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:52.666 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:21:52.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 -- 172.21.15.77:0/170933982 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ed006be30 msgr2=0x7f0ed006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ed006be30 0x7f0ed006e2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0eec158450 tx=0x7f0edc016040 comp rx=0 tx=0).stop 2024-04-25T01:21:52.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 -- 172.21.15.77:0/170933982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 msgr2=0x7f0eec153170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 0x7f0eec153170 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009ac0 tx=0x7f0ee8004730 comp rx=0 tx=0).stop 2024-04-25T01:21:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 -- 172.21.15.77:0/170933982 shutdown_connections 2024-04-25T01:21:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ed006be30 0x7f0ed006e2f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eec152440 0x7f0eec0aeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eec152c40 0x7f0eec06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 --2- 172.21.15.77:0/170933982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed80039b0 0x7f0eec153170 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 -- 172.21.15.77:0/170933982 >> 172.21.15.77:0/170933982 conn(0x7f0eec0d1c80 msgr2=0x7f0eec0afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 -- 172.21.15.77:0/170933982 shutdown_connections 2024-04-25T01:21:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:52.685+0000 7f0ef4038640 1 -- 172.21.15.77:0/170933982 wait complete. 2024-04-25T01:21:53.081 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:21:45.408154+0000', 'modified': '2024-04-25T01:21:50.465658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6835/2810316639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6834', 'nonce': 2810316639}, {'type': 'v1', 'addr': '172.21.15.121:6835', 'nonce': 2810316639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4294, 'qdb_cluster': [4294]} 2024-04-25T01:21:53.081 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:21:53.081 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T01:21:53.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.199+0000 7fa04df30640 1 Processor -- start 2024-04-25T01:21:53.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.199+0000 7fa04df30640 1 -- start start 2024-04-25T01:21:53.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.199+0000 7fa04df30640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa04806fcd0 0x7fa04815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.199+0000 7fa04df30640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0480b3560 con 0x7fa0480ade10 2024-04-25T01:21:53.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.199+0000 7fa04df30640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0480ae510 con 0x7fa04806c840 2024-04-25T01:21:53.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.199+0000 7fa04df30640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa0481589e0 con 0x7fa04806fcd0 2024-04-25T01:21:53.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa046ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa0480ade10 0x7fa04806c130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53888/0 (socket says 172.21.15.77:53888) 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa046ffd640 1 -- 172.21.15.77:0/1351778405 learned_addr learned my addr 172.21.15.77:0/1351778405 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2060789934 0 0) 0x7fa0480b3560 con 0x7fa0480ade10 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0477fe640 1 --2- 172.21.15.77:0/1351778405 >> v2:172.21.15.121:3300/0 conn(0x7fa04806fcd0 0x7fa04815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa030003520 con 0x7fa0480ade10 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2924790593 0 0) 0x7fa030003520 con 0x7fa0480ade10 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 >> v2:172.21.15.121:3300/0 conn(0x7fa04806fcd0 msgr2=0x7fa04815ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 --2- 172.21.15.77:0/1351778405 >> v2:172.21.15.121:3300/0 conn(0x7fa04806fcd0 0x7fa04815ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 >> v1:172.21.15.121:6789/0 conn(0x7fa04806c840 legacy=0x7fa0480b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0480ae7f0 con 0x7fa0480ade10 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fa03c001690 con 0x7fa0480ade10 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 >> v1:172.21.15.77:6789/0 conn(0x7fa0480ade10 legacy=0x7fa04806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 --2- 172.21.15.77:0/1351778405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0300039b0 0x7fa030003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 --2- 172.21.15.77:0/1351778405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa030004800 0x7fa030004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 --2- 172.21.15.77:0/1351778405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 0x7fa03000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481589e0 con 0x7fa0300051c0 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa03000c160 con 0x7fa0300039b0 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa03000c360 con 0x7fa030004800 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa047fff640 1 --2- 172.21.15.77:0/1351778405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 0x7fa03000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa0477fe640 1 --2- 172.21.15.77:0/1351778405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0300039b0 0x7fa030003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.200+0000 7fa046ffd640 1 --2- 172.21.15.77:0/1351778405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa030004800 0x7fa030004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.201+0000 7fa047fff640 1 -- 172.21.15.77:0/1351778405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa030004800 msgr2=0x7fa030004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.201+0000 7fa047fff640 1 --2- 172.21.15.77:0/1351778405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa030004800 0x7fa030004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.201+0000 7fa047fff640 1 -- 172.21.15.77:0/1351778405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0300039b0 msgr2=0x7fa030003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.201+0000 7fa047fff640 1 --2- 172.21.15.77:0/1351778405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0300039b0 0x7fa030003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.201+0000 7fa047fff640 1 -- 172.21.15.77:0/1351778405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa030003520 con 0x7fa0300051c0 2024-04-25T01:21:53.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.201+0000 7fa047fff640 1 -- 172.21.15.77:0/1351778405 reap_dead start 2024-04-25T01:21:53.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.201+0000 7fa047fff640 1 --2- 172.21.15.77:0/1351778405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 0x7fa03000b9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa040001720 tx=0x7fa030005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f856024dcc11bae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:53.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.201+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040001370 con 0x7fa0300051c0 2024-04-25T01:21:53.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.201+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0400052d0 con 0x7fa0300051c0 2024-04-25T01:21:53.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.201+0000 7fa0467fc640 1 -- 172.21.15.77:0/1351778405 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04002abc0 con 0x7fa0300051c0 2024-04-25T01:21:53.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.202+0000 7fa04df30640 1 -- 172.21.15.77:0/1351778405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 msgr2=0x7fa03000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.202+0000 7fa04df30640 1 --2- 172.21.15.77:0/1351778405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 0x7fa03000b9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa040001720 tx=0x7fa030005830 comp rx=0 tx=0).stop 2024-04-25T01:21:53.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.202+0000 7fa04df30640 1 -- 172.21.15.77:0/1351778405 shutdown_connections 2024-04-25T01:21:53.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.202+0000 7fa04df30640 1 --2- 172.21.15.77:0/1351778405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 0x7fa03000b9c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.202+0000 7fa04df30640 1 -- 172.21.15.77:0/1351778405 >> 172.21.15.77:0/1351778405 conn(0x7fa048071550 msgr2=0x7fa048071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:53.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.202+0000 7fa04df30640 1 -- 172.21.15.77:0/1351778405 shutdown_connections 2024-04-25T01:21:53.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.202+0000 7fa04df30640 1 -- 172.21.15.77:0/1351778405 wait complete. 2024-04-25T01:21:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.203+0000 7fa04df30640 1 Processor -- start 2024-04-25T01:21:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.203+0000 7fa04df30640 1 -- start start 2024-04-25T01:21:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa04df30640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 0x7fa048146b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa04df30640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa048158b60 0x7fa048143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa04df30640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0480ce880 0x7fa0480ced00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa04df30640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0480ae2c0 con 0x7fa0300051c0 2024-04-25T01:21:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa04df30640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa0480b3890 con 0x7fa0480ce880 2024-04-25T01:21:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa04df30640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa04815ef50 con 0x7fa048158b60 2024-04-25T01:21:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa0477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 0x7fa048146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa0477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 0x7fa048146b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40792/0 (socket says 172.21.15.77:40792) 2024-04-25T01:21:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa0477fe640 1 -- 172.21.15.77:0/1916653215 learned_addr learned my addr 172.21.15.77:0/1916653215 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa046ffd640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa048158b60 0x7fa048143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa047fff640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0480ce880 0x7fa0480ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa0477fe640 1 -- 172.21.15.77:0/1916653215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa048158b60 msgr2=0x7fa048143970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa0477fe640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa048158b60 0x7fa048143970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa0477fe640 1 -- 172.21.15.77:0/1916653215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0480ce880 msgr2=0x7fa0480ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa0477fe640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0480ce880 0x7fa0480ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa0477fe640 1 -- 172.21.15.77:0/1916653215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa048143eb0 con 0x7fa0300051c0 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa0477fe640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 0x7fa048146b30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa038009f90 tx=0x7fa03800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.204+0000 7fa02ffff640 1 -- 172.21.15.77:0/1916653215 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038015430 con 0x7fa0300051c0 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.205+0000 7fa02ffff640 1 -- 172.21.15.77:0/1916653215 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0380155d0 con 0x7fa0300051c0 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.205+0000 7fa04df30640 1 -- 172.21.15.77:0/1916653215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa048157570 con 0x7fa0300051c0 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.205+0000 7fa02ffff640 1 -- 172.21.15.77:0/1916653215 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0380159c0 con 0x7fa0300051c0 2024-04-25T01:21:53.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.205+0000 7fa04df30640 1 -- 172.21.15.77:0/1916653215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa048144230 con 0x7fa0300051c0 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.206+0000 7fa02ffff640 1 -- 172.21.15.77:0/1916653215 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa038015b60 con 0x7fa0300051c0 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.206+0000 7fa02ffff640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa03406be60 0x7fa03406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.206+0000 7fa046ffd640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa03406be60 0x7fa03406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.206+0000 7fa02ffff640 1 -- 172.21.15.77:0/1916653215 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa03808c2a0 con 0x7fa0300051c0 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.207+0000 7fa046ffd640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa03406be60 0x7fa03406e320 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa0481e7ea0 tx=0x7fa03c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.207+0000 7fa050c3f740 1 Processor -- start 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.207+0000 7fa050c3f740 1 -- start start 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.208+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2f7e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.208+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2faec80 0x55c0d2fae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.208+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fae540 0x55c0d2f83350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.208+0000 7fa050c3f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c0d2ee6f00 con 0x55c0d2fae540 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.208+0000 7fa050c3f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c0d2e8f750 con 0x55c0d2faec80 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.208+0000 7fa050c3f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c0d2ee2ec0 con 0x55c0d2fcbb60 2024-04-25T01:21:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.208+0000 7fa047fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fae540 0x55c0d2f83350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.208+0000 7fa047fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fae540 0x55c0d2f83350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40800/0 (socket says 172.21.15.77:40800) 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.208+0000 7fa047fff640 1 -- 172.21.15.77:0/2361801314 learned_addr learned my addr 172.21.15.77:0/2361801314 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.208+0000 7fa0477fe640 1 --2- 172.21.15.77:0/2361801314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2f7e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.208+0000 7fa046ffd640 1 --2- 172.21.15.77:0/2361801314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2faec80 0x55c0d2fae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.209+0000 7fa047fff640 1 -- 172.21.15.77:0/2361801314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 msgr2=0x55c0d2f7e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.209+0000 7fa047fff640 1 --2- 172.21.15.77:0/2361801314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2f7e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.209+0000 7fa047fff640 1 -- 172.21.15.77:0/2361801314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2faec80 msgr2=0x55c0d2fae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.209+0000 7fa047fff640 1 --2- 172.21.15.77:0/2361801314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2faec80 0x55c0d2fae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.209+0000 7fa047fff640 1 -- 172.21.15.77:0/2361801314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0d2bfb2d0 con 0x55c0d2fae540 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.209+0000 7fa047fff640 1 --2- 172.21.15.77:0/2361801314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fae540 0x55c0d2f83350 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa04002a570 tx=0x7fa04002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15dae97456be713f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.209+0000 7fa02d7fa640 1 -- 172.21.15.77:0/2361801314 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0400052d0 con 0x55c0d2fae540 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.209+0000 7fa02d7fa640 1 -- 172.21.15.77:0/2361801314 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa040005470 con 0x55c0d2fae540 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.209+0000 7fa02d7fa640 1 -- 172.21.15.77:0/2361801314 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04002aba0 con 0x55c0d2fae540 2024-04-25T01:21:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.210+0000 7fa050c3f740 1 -- 172.21.15.77:0/2361801314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fae540 msgr2=0x55c0d2f83350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.210+0000 7fa050c3f740 1 --2- 172.21.15.77:0/2361801314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fae540 0x55c0d2f83350 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa04002a570 tx=0x7fa04002e000 comp rx=0 tx=0).stop 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.210+0000 7fa050c3f740 1 -- 172.21.15.77:0/2361801314 shutdown_connections 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.210+0000 7fa050c3f740 1 --2- 172.21.15.77:0/2361801314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fae540 0x55c0d2f83350 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.210+0000 7fa050c3f740 1 --2- 172.21.15.77:0/2361801314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2faec80 0x55c0d2fae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.210+0000 7fa050c3f740 1 --2- 172.21.15.77:0/2361801314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2f7e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.210+0000 7fa050c3f740 1 -- 172.21.15.77:0/2361801314 >> 172.21.15.77:0/2361801314 conn(0x55c0d2f80760 msgr2=0x55c0d2fc9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.210+0000 7fa050c3f740 1 -- 172.21.15.77:0/2361801314 shutdown_connections 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.210+0000 7fa050c3f740 1 -- 172.21.15.77:0/2361801314 wait complete. 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.210+0000 7fa050c3f740 1 -- start start 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.210+0000 7fa050c3f740 1 Processor -- start 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2faec80 0x55c0d2f83270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2fcbb60 0x55c0d2f837b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fd8b40 0x55c0d2fc0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa050c3f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c0d2f82910 con 0x55c0d2fd8b40 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa050c3f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c0d2ea2cf0 con 0x55c0d2fcbb60 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa050c3f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c0d2f81d00 con 0x55c0d2faec80 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa050c3f740 10 client.0 _ll_register_callbacks cb 0x55c0d2ea26e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa047fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fd8b40 0x55c0d2fc0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa047fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fd8b40 0x55c0d2fc0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40814/0 (socket says 172.21.15.77:40814) 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa047fff640 1 -- 172.21.15.77:0/1389079545 learned_addr learned my addr 172.21.15.77:0/1389079545 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa046ffd640 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2fcbb60 0x55c0d2f837b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa0477fe640 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2faec80 0x55c0d2f83270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa047fff640 1 -- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2faec80 msgr2=0x55c0d2f83270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa047fff640 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2faec80 0x55c0d2f83270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa047fff640 1 -- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2fcbb60 msgr2=0x55c0d2f837b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa047fff640 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2fcbb60 0x55c0d2f837b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa047fff640 1 -- 172.21.15.77:0/1389079545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0d2cc56b0 con 0x55c0d2fd8b40 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa046ffd640 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2fcbb60 0x55c0d2f837b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.211+0000 7fa0477fe640 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2faec80 0x55c0d2f83270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa047fff640 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fd8b40 0x55c0d2fc0fb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa040007ba0 tx=0x7fa040001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875b14133b9bdce0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa0267fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa0267fc640 1 -- 172.21.15.77:0/1389079545 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04002ab90 con 0x55c0d2fd8b40 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa0267fc640 1 -- 172.21.15.77:0/1389079545 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa04002ad30 con 0x55c0d2fd8b40 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa050c3f740 1 -- 172.21.15.77:0/1389079545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa040005470 con 0x55c0d2fd8b40 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa0267fc640 1 -- 172.21.15.77:0/1389079545 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fa04004e050 con 0x55c0d2fd8b40 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa050c3f740 10 client.4413 fetch_fsmap learned FSMap version 7 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa050c3f740 1 -- 172.21.15.77:0/1389079545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55c0d2fd8f80 con 0x55c0d2fd8b40 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa0267fc640 1 -- 172.21.15.77:0/1389079545 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa040005470 con 0x55c0d2fd8b40 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa0267fc640 1 -- 172.21.15.77:0/1389079545 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7fa040007020 con 0x55c0d2fd8b40 2024-04-25T01:21:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa050c3f740 10 client.4413 fetch_fsmap finished waiting for FSMap version 7 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa050c3f740 10 client.4413 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa050c3f740 20 client.4413 populate_metadata read hostname 'smithi077' 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa050c3f740 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x55c0d2fa9280 0x55c0d2faf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa050c3f740 4 client.4413 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa050c3f740 1 -- 172.21.15.77:0/1389079545 --> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0d2ef4440 con 0x55c0d2fa9280 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.212+0000 7fa0477fe640 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x55c0d2fa9280 0x55c0d2faf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.213+0000 7fa0267fc640 10 client.4413 ms_handle_connect on v2:172.21.15.121:6834/2810316639 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.213+0000 7fa0477fe640 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x55c0d2fa9280 0x55c0d2faf240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a160ea673e01e19 server_cookie=fb4b38ac21c27b44 in_seq=0 out_seq=0 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.213+0000 7fa0267fc640 1 -- 172.21.15.77:0/1389079545 <== mds.0 v2:172.21.15.121:6834/2810316639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0d2ef4440 con 0x55c0d2fa9280 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.213+0000 7fa0267fc640 10 client.4413 handle_command_reply: tid=0 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.213+0000 7fa050c3f740 1 client.4413 shutdown 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa050c3f740 1 -- 172.21.15.77:0/1389079545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fd8b40 msgr2=0x55c0d2fc0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa050c3f740 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fd8b40 0x55c0d2fc0fb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa040007ba0 tx=0x7fa040001190 comp rx=0 tx=0).stop 2024-04-25T01:21:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa050c3f740 1 -- 172.21.15.77:0/1389079545 shutdown_connections 2024-04-25T01:21:53.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa050c3f740 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x55c0d2fa9280 0x55c0d2faf240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa050c3f740 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fd8b40 0x55c0d2fc0fb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa050c3f740 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2fcbb60 0x55c0d2f837b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa0267fc640 0 client.4413 ms_handle_reset on v2:172.21.15.121:6834/2810316639 2024-04-25T01:21:53.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa050c3f740 1 --2- 172.21.15.77:0/1389079545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2faec80 0x55c0d2f83270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa050c3f740 1 -- 172.21.15.77:0/1389079545 >> 172.21.15.77:0/1389079545 conn(0x55c0d2fae540 msgr2=0x55c0d2fc9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:53.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa050c3f740 1 -- 172.21.15.77:0/1389079545 shutdown_connections 2024-04-25T01:21:53.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa050c3f740 1 -- 172.21.15.77:0/1389079545 wait complete. 2024-04-25T01:21:53.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.214+0000 7fa050c3f740 20 client.4413 trim_cache size 0 max 16384 2024-04-25T01:21:53.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.231+0000 7fa050c3f740 1 Processor -- start 2024-04-25T01:21:53.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.231+0000 7fa050c3f740 1 -- start start 2024-04-25T01:21:53.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.232+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2fa9280 0x55c0d2fec940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.232+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2faec80 0x55c0d2fc0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.232+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2fc8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.232+0000 7fa050c3f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa048201200 con 0x55c0d2faec80 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.232+0000 7fa050c3f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa048200770 con 0x55c0d2fa9280 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.232+0000 7fa050c3f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa048200c10 con 0x55c0d2fcbb60 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.232+0000 7fa046ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2faec80 0x55c0d2fc0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.232+0000 7fa046ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2faec80 0x55c0d2fc0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40822/0 (socket says 172.21.15.77:40822) 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.232+0000 7fa046ffd640 1 -- 172.21.15.77:0/1236479527 learned_addr learned my addr 172.21.15.77:0/1236479527 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa047fff640 1 --2- 172.21.15.77:0/1236479527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2fc8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa0477fe640 1 --2- 172.21.15.77:0/1236479527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2fa9280 0x55c0d2fec940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa046ffd640 1 -- 172.21.15.77:0/1236479527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 msgr2=0x55c0d2fc8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa046ffd640 1 --2- 172.21.15.77:0/1236479527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2fc8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa046ffd640 1 -- 172.21.15.77:0/1236479527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2fa9280 msgr2=0x55c0d2fec940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa046ffd640 1 --2- 172.21.15.77:0/1236479527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2fa9280 0x55c0d2fec940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa046ffd640 1 -- 172.21.15.77:0/1236479527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0d2fb1630 con 0x55c0d2faec80 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa046ffd640 1 --2- 172.21.15.77:0/1236479527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2faec80 0x55c0d2fc0f90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa03c0015c0 tx=0x7fa03c00c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557e1c853f0ded62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa02cff9640 1 -- 172.21.15.77:0/1236479527 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c024420 con 0x55c0d2faec80 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa02cff9640 1 -- 172.21.15.77:0/1236479527 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa03c029020 con 0x55c0d2faec80 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa02cff9640 1 -- 172.21.15.77:0/1236479527 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c0245c0 con 0x55c0d2faec80 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.233+0000 7fa050c3f740 1 -- 172.21.15.77:0/1236479527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2faec80 msgr2=0x55c0d2fc0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.234+0000 7fa050c3f740 1 --2- 172.21.15.77:0/1236479527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2faec80 0x55c0d2fc0f90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa03c0015c0 tx=0x7fa03c00c000 comp rx=0 tx=0).stop 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.234+0000 7fa050c3f740 1 -- 172.21.15.77:0/1236479527 shutdown_connections 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.234+0000 7fa050c3f740 1 --2- 172.21.15.77:0/1236479527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2faec80 0x55c0d2fc0f90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.234+0000 7fa050c3f740 1 --2- 172.21.15.77:0/1236479527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2fc8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.234+0000 7fa050c3f740 1 --2- 172.21.15.77:0/1236479527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2fa9280 0x55c0d2fec940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.234+0000 7fa050c3f740 1 -- 172.21.15.77:0/1236479527 >> 172.21.15.77:0/1236479527 conn(0x55c0d2fae540 msgr2=0x55c0d2f79690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.234+0000 7fa050c3f740 1 -- 172.21.15.77:0/1236479527 shutdown_connections 2024-04-25T01:21:53.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.234+0000 7fa050c3f740 1 -- 172.21.15.77:0/1236479527 wait complete. 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.234+0000 7fa050c3f740 1 -- start start 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.234+0000 7fa050c3f740 1 Processor -- start 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fa9280 0x55c0d2f7cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2faec80 0x55c0d2f7d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa050c3f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2fcc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa050c3f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c0d2f82910 con 0x55c0d2fa9280 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa050c3f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c0d2ea2cf0 con 0x55c0d2faec80 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa050c3f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c0d2fa98a0 con 0x55c0d2fcbb60 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa050c3f740 10 client.0 _ll_register_callbacks cb 0x7fa0380095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa0477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fa9280 0x55c0d2f7cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa0477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fa9280 0x55c0d2f7cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40830/0 (socket says 172.21.15.77:40830) 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa0477fe640 1 -- 172.21.15.77:0/3978783309 learned_addr learned my addr 172.21.15.77:0/3978783309 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa046ffd640 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2faec80 0x55c0d2f7d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa047fff640 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2fcc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa0477fe640 1 -- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 msgr2=0x55c0d2fcc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa0477fe640 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2fcc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa0477fe640 1 -- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2faec80 msgr2=0x55c0d2f7d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa0477fe640 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2faec80 0x55c0d2f7d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa0477fe640 1 -- 172.21.15.77:0/3978783309 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0d2fec290 con 0x55c0d2fa9280 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa047fff640 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2fcc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa046ffd640 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2faec80 0x55c0d2f7d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa0477fe640 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fa9280 0x55c0d2f7cf70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa038009f60 tx=0x7fa038056690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cda2a07fa929d8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.235+0000 7fa0267fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa0267fc640 1 -- 172.21.15.77:0/3978783309 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03808a060 con 0x55c0d2fa9280 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa0267fc640 1 -- 172.21.15.77:0/3978783309 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa038072130 con 0x55c0d2fa9280 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa050c3f740 1 -- 172.21.15.77:0/3978783309 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa03c029020 con 0x55c0d2fa9280 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa0267fc640 1 -- 172.21.15.77:0/3978783309 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fa038078050 con 0x55c0d2fa9280 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa050c3f740 10 client.4419 fetch_fsmap learned FSMap version 7 2024-04-25T01:21:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa050c3f740 1 -- 172.21.15.77:0/3978783309 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55c0d2fe1540 con 0x55c0d2fa9280 2024-04-25T01:21:53.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa0267fc640 1 -- 172.21.15.77:0/3978783309 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa03c029020 con 0x55c0d2fa9280 2024-04-25T01:21:53.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa0267fc640 1 -- 172.21.15.77:0/3978783309 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7fa038025030 con 0x55c0d2fa9280 2024-04-25T01:21:53.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa050c3f740 10 client.4419 fetch_fsmap finished waiting for FSMap version 7 2024-04-25T01:21:53.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa050c3f740 10 client.4419 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T01:21:53.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa050c3f740 20 client.4419 populate_metadata read hostname 'smithi077' 2024-04-25T01:21:53.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa050c3f740 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x55c0d2fd8b40 0x55c0d304acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa050c3f740 4 client.4419 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:21:53.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa050c3f740 1 -- 172.21.15.77:0/3978783309 --> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] -- command(tid 0: {"prefix": "status"}) -- 0x55c0d2eee790 con 0x55c0d2fd8b40 2024-04-25T01:21:53.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.236+0000 7fa046ffd640 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x55c0d2fd8b40 0x55c0d304acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa0267fc640 10 client.4419 ms_handle_connect on v2:172.21.15.121:6834/2810316639 2024-04-25T01:21:53.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa046ffd640 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x55c0d2fd8b40 0x55c0d304acc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e886fbf9c9f94345 server_cookie=11f9aa4a744fad20 in_seq=0 out_seq=0 2024-04-25T01:21:53.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa0267fc640 1 -- 172.21.15.77:0/3978783309 <== mds.0 v2:172.21.15.121:6834/2810316639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55c0d2eee790 con 0x55c0d2fd8b40 2024-04-25T01:21:53.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa0267fc640 10 client.4419 handle_command_reply: tid=0 2024-04-25T01:21:53.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 1 client.4419 shutdown 2024-04-25T01:21:53.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 1 -- 172.21.15.77:0/3978783309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fa9280 msgr2=0x55c0d2f7cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fa9280 0x55c0d2f7cf70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa038009f60 tx=0x7fa038056690 comp rx=0 tx=0).stop 2024-04-25T01:21:53.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 1 -- 172.21.15.77:0/3978783309 shutdown_connections 2024-04-25T01:21:53.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x55c0d2fd8b40 0x55c0d304acc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c0d2fcbb60 0x55c0d2fcc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0d2faec80 0x55c0d2f7d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa0267fc640 0 client.4419 ms_handle_reset on v2:172.21.15.121:6834/2810316639 2024-04-25T01:21:53.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 1 --2- 172.21.15.77:0/3978783309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0d2fa9280 0x55c0d2f7cf70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 1 -- 172.21.15.77:0/3978783309 >> 172.21.15.77:0/3978783309 conn(0x55c0d2fae540 msgr2=0x55c0d2f7e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:53.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 1 -- 172.21.15.77:0/3978783309 shutdown_connections 2024-04-25T01:21:53.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 1 -- 172.21.15.77:0/3978783309 wait complete. 2024-04-25T01:21:53.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa050c3f740 20 client.4419 trim_cache size 0 max 16384 2024-04-25T01:21:53.229 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:21:53.229 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "7c3c39ee-077e-476c-8994-eede4fab301d", 2024-04-25T01:21:53.229 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-04-25T01:21:53.230 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 4294, 2024-04-25T01:21:53.230 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-04-25T01:21:53.230 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-04-25T01:21:53.230 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-04-25T01:21:53.230 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 7.1660348110000003, 2024-04-25T01:21:53.230 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 7, 2024-04-25T01:21:53.230 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 24, 2024-04-25T01:21:53.230 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 23, 2024-04-25T01:21:53.230 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 14.408531026 2024-04-25T01:21:53.230 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:21:53.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa04df30640 1 -- 172.21.15.77:0/1916653215 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa03406be60 msgr2=0x7fa03406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa04df30640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa03406be60 0x7fa03406e320 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa0481e7ea0 tx=0x7fa03c009040 comp rx=0 tx=0).stop 2024-04-25T01:21:53.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa04df30640 1 -- 172.21.15.77:0/1916653215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 msgr2=0x7fa048146b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa04df30640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 0x7fa048146b30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa038009f90 tx=0x7fa03800f580 comp rx=0 tx=0).stop 2024-04-25T01:21:53.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa04df30640 1 -- 172.21.15.77:0/1916653215 shutdown_connections 2024-04-25T01:21:53.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa04df30640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa03406be60 0x7fa03406e320 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa04df30640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0480ce880 0x7fa0480ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa04df30640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa048158b60 0x7fa048143970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.237+0000 7fa04df30640 1 --2- 172.21.15.77:0/1916653215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0300051c0 0x7fa048146b30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.238+0000 7fa04df30640 1 -- 172.21.15.77:0/1916653215 >> 172.21.15.77:0/1916653215 conn(0x7fa048071550 msgr2=0x7fa0480b0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:53.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.238+0000 7fa04df30640 1 -- 172.21.15.77:0/1916653215 shutdown_connections 2024-04-25T01:21:53.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.238+0000 7fa04df30640 1 -- 172.21.15.77:0/1916653215 wait complete. 2024-04-25T01:21:53.233 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:21:53.233 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:21:53.234 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:21:53.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.349+0000 7fd94eddc640 1 Processor -- start 2024-04-25T01:21:53.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.349+0000 7fd94eddc640 1 -- start start 2024-04-25T01:21:53.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.350+0000 7fd94eddc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd94806b120 0x7fd9480afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.350+0000 7fd94eddc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd9480cea60 0x7fd948072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.350+0000 7fd94eddc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9480b5430 con 0x7fd9480d1760 2024-04-25T01:21:53.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.350+0000 7fd94eddc640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fd948071d80 con 0x7fd9480cea60 2024-04-25T01:21:53.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.350+0000 7fd94eddc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd9480b0300 con 0x7fd94806b120 2024-04-25T01:21:53.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.350+0000 7fd94d379640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd9480cea60 0x7fd948072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.350+0000 7fd947fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd9480d1760 0x7fd94806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53890/0 (socket says 172.21.15.77:53890) 2024-04-25T01:21:53.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.350+0000 7fd947fff640 1 -- 172.21.15.77:0/215833312 learned_addr learned my addr 172.21.15.77:0/215833312 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.350+0000 7fd94cb78640 1 --2- 172.21.15.77:0/215833312 >> v2:172.21.15.121:3300/0 conn(0x7fd94806b120 0x7fd9480afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.351+0000 7fd94d379640 1 -- 172.21.15.77:0/215833312 >> v2:172.21.15.121:3300/0 conn(0x7fd94806b120 msgr2=0x7fd9480afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.351+0000 7fd94d379640 1 --2- 172.21.15.77:0/215833312 >> v2:172.21.15.121:3300/0 conn(0x7fd94806b120 0x7fd9480afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.351+0000 7fd94d379640 1 -- 172.21.15.77:0/215833312 >> v1:172.21.15.77:6789/0 conn(0x7fd9480d1760 legacy=0x7fd94806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.351+0000 7fd94d379640 1 -- 172.21.15.77:0/215833312 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9480b5710 con 0x7fd9480cea60 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.351+0000 7fd94d379640 1 --2- 172.21.15.77:0/215833312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9480cea60 0x7fd948072210 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd9480b4240 tx=0x7fd94002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8da06758057cfdc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.351+0000 7fd9477fe640 1 -- 172.21.15.77:0/215833312 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940007740 con 0x7fd9480cea60 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.351+0000 7fd9477fe640 1 -- 172.21.15.77:0/215833312 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9400078e0 con 0x7fd9480cea60 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.351+0000 7fd9477fe640 1 -- 172.21.15.77:0/215833312 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940007cb0 con 0x7fd9480cea60 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.352+0000 7fd94eddc640 1 -- 172.21.15.77:0/215833312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9480cea60 msgr2=0x7fd948072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.352+0000 7fd94eddc640 1 --2- 172.21.15.77:0/215833312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9480cea60 0x7fd948072210 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd9480b4240 tx=0x7fd94002a5f0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.352+0000 7fd94eddc640 1 -- 172.21.15.77:0/215833312 shutdown_connections 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.352+0000 7fd94eddc640 1 --2- 172.21.15.77:0/215833312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9480cea60 0x7fd948072210 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.352+0000 7fd94eddc640 1 --2- 172.21.15.77:0/215833312 >> v2:172.21.15.121:3300/0 conn(0x7fd94806b120 0x7fd9480afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.352+0000 7fd94eddc640 1 -- 172.21.15.77:0/215833312 >> 172.21.15.77:0/215833312 conn(0x7fd9480d0b60 msgr2=0x7fd9480d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:53.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.352+0000 7fd94eddc640 1 -- 172.21.15.77:0/215833312 shutdown_connections 2024-04-25T01:21:53.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.352+0000 7fd94eddc640 1 -- 172.21.15.77:0/215833312 wait complete. 2024-04-25T01:21:53.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.353+0000 7fd94eddc640 1 Processor -- start 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.353+0000 7fd94eddc640 1 -- start start 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94eddc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd94806b120 0x7fd94815a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94eddc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9480cea60 0x7fd9481e7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94eddc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9480d1760 0x7fd948160a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94eddc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd94815e530 con 0x7fd9480d1760 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94eddc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd9480b5cd0 con 0x7fd9480cea60 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94eddc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd94815e6b0 con 0x7fd94806b120 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94d379640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9480d1760 0x7fd948160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94d379640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9480d1760 0x7fd948160a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40854/0 (socket says 172.21.15.77:40854) 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94d379640 1 -- 172.21.15.77:0/1070683404 learned_addr learned my addr 172.21.15.77:0/1070683404 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94d379640 1 -- 172.21.15.77:0/1070683404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd94806b120 msgr2=0x7fd94815a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94d379640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd94806b120 0x7fd94815a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94d379640 1 -- 172.21.15.77:0/1070683404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9480cea60 msgr2=0x7fd9481e7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94d379640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9480cea60 0x7fd9481e7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94d379640 1 -- 172.21.15.77:0/1070683404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94815f5e0 con 0x7fd9480d1760 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.354+0000 7fd94d379640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9480d1760 0x7fd948160a70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd94002abe0 tx=0x7fd940004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.355+0000 7fd9457fa640 1 -- 172.21.15.77:0/1070683404 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940040070 con 0x7fd9480d1760 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.355+0000 7fd9457fa640 1 -- 172.21.15.77:0/1070683404 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd940031bc0 con 0x7fd9480d1760 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.355+0000 7fd94eddc640 1 -- 172.21.15.77:0/1070683404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd94815f8c0 con 0x7fd9480d1760 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.355+0000 7fd9457fa640 1 -- 172.21.15.77:0/1070683404 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940045460 con 0x7fd9480d1760 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.355+0000 7fd94eddc640 1 -- 172.21.15.77:0/1070683404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd948160160 con 0x7fd9480d1760 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.356+0000 7fd94eddc640 1 -- 172.21.15.77:0/1070683404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd94815fcb0 con 0x7fd9480d1760 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.356+0000 7fd9457fa640 1 -- 172.21.15.77:0/1070683404 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd940039040 con 0x7fd9480d1760 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.356+0000 7fd9457fa640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd93406be60 0x7fd93406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.356+0000 7fd9457fa640 1 -- 172.21.15.77:0/1070683404 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd9400aca80 con 0x7fd9480d1760 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.356+0000 7fd94cb78640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd93406be60 0x7fd93406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.357+0000 7fd94cb78640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd93406be60 0x7fd93406e320 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd9380098f0 tx=0x7fd938008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:53.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.361+0000 7fd9457fa640 1 -- 172.21.15.77:0/1070683404 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd94007b0f0 con 0x7fd9480d1760 2024-04-25T01:21:53.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.571+0000 7fd94eddc640 1 -- 172.21.15.77:0/1070683404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd948160360 con 0x7fd9480d1760 2024-04-25T01:21:53.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.572+0000 7fd9457fa640 1 -- 172.21.15.77:0/1070683404 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7fd948160360 con 0x7fd9480d1760 2024-04-25T01:21:53.553 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.573+0000 7fd94eddc640 1 -- 172.21.15.77:0/1070683404 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd93406be60 msgr2=0x7fd93406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.573+0000 7fd94eddc640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd93406be60 0x7fd93406e320 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd9380098f0 tx=0x7fd938008040 comp rx=0 tx=0).stop 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.573+0000 7fd94eddc640 1 -- 172.21.15.77:0/1070683404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9480d1760 msgr2=0x7fd948160a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.573+0000 7fd94eddc640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9480d1760 0x7fd948160a70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd94002abe0 tx=0x7fd940004a40 comp rx=0 tx=0).stop 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.574+0000 7fd94eddc640 1 -- 172.21.15.77:0/1070683404 shutdown_connections 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.574+0000 7fd94eddc640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd93406be60 0x7fd93406e320 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.574+0000 7fd94eddc640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9480d1760 0x7fd948160a70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.574+0000 7fd94eddc640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9480cea60 0x7fd9481e7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.574+0000 7fd94eddc640 1 --2- 172.21.15.77:0/1070683404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd94806b120 0x7fd94815a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.574+0000 7fd94eddc640 1 -- 172.21.15.77:0/1070683404 >> 172.21.15.77:0/1070683404 conn(0x7fd9480d0b60 msgr2=0x7fd9480b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.574+0000 7fd94eddc640 1 -- 172.21.15.77:0/1070683404 shutdown_connections 2024-04-25T01:21:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53.574+0000 7fd94eddc640 1 -- 172.21.15.77:0/1070683404 wait complete. 2024-04-25T01:21:53.566 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T01:21:53.566 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:21:53.566 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-04-25T01:21:53.566 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-25T01:21:53.566 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:21:53.566 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T01:21:53.566 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:21:53.566 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:21:53.566 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T01:21:53.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:21:53.567 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-04-25T01:21:53.581 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:21:53.581 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:21:53.581 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:21:53.582 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:21:53.583 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T01:21:53.583 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42735sec preferred_lft 42735sec 2024-04-25T01:21:53.583 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a9ad:d2f:d456:e65f/64 scope link noprefixroute 2024-04-25T01:21:53.583 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:21:53.583 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:21:53.583 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:21:53.583 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:21:53.584 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:21:53.584 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr flush dev ceph-brx 2024-04-25T01:21:53.584 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set ceph-brx up 2024-04-25T01:21:53.584 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:21:53.584 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:21:53.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:21:53.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:21:53.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:21:53.764 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:21:53.787 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-04-25T01:21:53.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:21:53.789 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-04-25T01:21:53.856 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-04-25T01:21:53.856 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:21:53.856 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:21:53.856 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:21:53.857 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:21:53.857 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:21:53.858 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:21:53.858 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:21:53.858 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:21:53.858 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:21:53.858 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:21:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:21:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:21:54.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:21:54.111 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-04-25T01:21:54.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:21:54.126 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-04-25T01:21:54.180 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:21:54.180 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:21:54.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:21:54.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-25T01:21:54.181 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:21:54.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:21:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:21:54.295 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-25T01:21:54.295 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:21:54.295 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:21:54.295 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-25T01:21:54.295 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:21:54.295 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-25T01:21:54.295 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-25T01:21:54.295 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-25T01:21:54.295 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:21:54.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:21:54.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:21:54.435 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:21:54.436 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:21:54.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-04-25T01:21:54.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:21:54.436 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:21:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:21:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:21:54.510 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-25T01:21:54.511 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:21:54.511 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:21:54.524 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T01:21:54.524 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:21:54.525 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:21:54.579 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-04-25T01:21:54.643 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:21:54.699 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-04-25T01:21:54.699 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-04-25T01:21:54.699 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-04-25T01:21:54.699 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-04-25T01:21:54.699 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-04-25T01:21:54.699 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-04-25T01:21:54.699 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:21:54.700 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:21:54.701 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-04-25T01:21:54.701 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-04-25T01:21:54.701 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T01:21:54.701 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T01:21:54.701 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T01:21:54.701 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T01:21:54.701 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-04-25T01:21:54.701 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:21:54.701 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:21:54.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:21:54.701 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-04-25T01:21:54.754 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T01:21:54.754 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-25T01:21:54.796 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-04-25T01:21:54.818 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:21:54.873 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-04-25T01:21:54.874 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-04-25T01:21:54.874 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-04-25T01:21:54.874 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-04-25T01:21:54.874 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-04-25T01:21:54.874 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-04-25T01:21:54.874 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-04-25T01:21:54.874 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-04-25T01:21:54.874 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-04-25T01:21:54.874 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-04-25T01:21:54.874 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-04-25T01:21:54.874 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-04-25T01:21:54.875 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T01:21:54.876 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T01:21:54.876 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T01:21:54.876 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T01:21:54.876 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-04-25T01:21:54.876 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:21:54.876 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:21:54.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:21:54.876 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-04-25T01:21:54.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi077.stderr:2024-04-25T01:21:54.919+0000 7f07acc14480 -1 init, newargv = 0x564c3bd10f30 newargc=15 2024-04-25T01:21:54.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi077.stderr:ceph-fuse[41981]: starting ceph client 2024-04-25T01:21:54.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi077.stderr:ceph-fuse[41981]: starting fuse 2024-04-25T01:21:54.914 INFO:teuthology.orchestra.run.smithi077.stdout:43 2024-04-25T01:21:54.915 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T01:21:54.915 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-04-25T01:21:54.915 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-04-25T01:21:54.915 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-04-25T01:21:54.915 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-04-25T01:21:54.915 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-04-25T01:21:54.915 DEBUG:teuthology.orchestra.run.smithi077:> 2024-04-25T01:21:54.915 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-04-25T01:21:54.915 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-25T01:21:54.915 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-04-25T01:21:54.915 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> 2024-04-25T01:21:54.916 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.admin")) 2024-04-25T01:21:54.917 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-04-25T01:21:55.031 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.admin.41981.asok 2024-04-25T01:21:55.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:21:55.041 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.41981.asok 2024-04-25T01:21:55.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:21:55.042 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.41981.asok status 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "admin", 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "41981", 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-04-25T01:21:55.175 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 4434, 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 4434 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3469123828 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-04-25T01:21:55.176 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3469123828 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.4434 192.168.0.1:0/3469123828", 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/3469123828", 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 7, 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 24, 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-04-25T01:21:55.177 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:21:55.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:21:55.183 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-04-25T01:21:55.183 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:21:55.198 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-04-25T01:21:55.198 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:21:55.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:21:55.198 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:21:55.263 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-25T01:21:55.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:21:55.369 INFO:teuthology.orchestra.run.smithi077.stdout:# file: . 2024-04-25T01:21:55.369 INFO:teuthology.orchestra.run.smithi077.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-25T01:21:55.369 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:55.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:21:55.375 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:55.375 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:21:55.390 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.543+0000 7f00803a2640 1 Processor -- start 2024-04-25T01:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.543+0000 7f00803a2640 1 -- start start 2024-04-25T01:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.544+0000 7f00803a2640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0078143940 0x7f007806e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.544+0000 7f00803a2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 0x7f007806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.544+0000 7f00803a2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0078070e60 con 0x7f007815ecf0 2024-04-25T01:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.544+0000 7f00803a2640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0078071140 con 0x7f0078143940 2024-04-25T01:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.544+0000 7f00803a2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0078071420 con 0x7f0078143d20 2024-04-25T01:21:55.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.544+0000 7f007d93d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 0x7f007806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.544+0000 7f007e93f640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f007815ecf0 0x7f007815f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:52584/0 (socket says 172.21.15.77:52584) 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.544+0000 7f007e13e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0078143940 0x7f007806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.544+0000 7f007e93f640 1 -- 172.21.15.77:0/3855559865 learned_addr learned my addr 172.21.15.77:0/3855559865 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.545+0000 7f007d13c640 1 -- 172.21.15.77:0/3855559865 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2431849361 0 0) 0x7f0078070e60 con 0x7f007815ecf0 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.545+0000 7f007d13c640 1 -- 172.21.15.77:0/3855559865 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0064003520 con 0x7f007815ecf0 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.545+0000 7f007d93d640 1 -- 172.21.15.77:0/3855559865 >> v2:172.21.15.121:3300/0 conn(0x7f0078143940 msgr2=0x7f007806e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.545+0000 7f007d93d640 1 --2- 172.21.15.77:0/3855559865 >> v2:172.21.15.121:3300/0 conn(0x7f0078143940 0x7f007806e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.545+0000 7f007d93d640 1 -- 172.21.15.77:0/3855559865 >> v1:172.21.15.121:6789/0 conn(0x7f007815ecf0 legacy=0x7f007815f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.545+0000 7f007d93d640 1 -- 172.21.15.77:0/3855559865 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0078071700 con 0x7f0078143d20 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f007d93d640 1 --2- 172.21.15.77:0/3855559865 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 0x7f007806eb90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f00680097a0 tx=0x7f006802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d4a53f360747d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f007d13c640 1 -- 172.21.15.77:0/3855559865 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0068037070 con 0x7f0078143d20 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f007d13c640 1 -- 172.21.15.77:0/3855559865 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0068032440 con 0x7f0078143d20 2024-04-25T01:21:55.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f007d13c640 1 -- 172.21.15.77:0/3855559865 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00680327f0 con 0x7f0078143d20 2024-04-25T01:21:55.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f00803a2640 1 -- 172.21.15.77:0/3855559865 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 msgr2=0x7f007806eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f00803a2640 1 --2- 172.21.15.77:0/3855559865 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 0x7f007806eb90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f00680097a0 tx=0x7f006802a180 comp rx=0 tx=0).stop 2024-04-25T01:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f00803a2640 1 -- 172.21.15.77:0/3855559865 shutdown_connections 2024-04-25T01:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f00803a2640 1 --2- 172.21.15.77:0/3855559865 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 0x7f007806eb90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f00803a2640 1 --2- 172.21.15.77:0/3855559865 >> v2:172.21.15.121:3300/0 conn(0x7f0078143940 0x7f007806e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f00803a2640 1 -- 172.21.15.77:0/3855559865 >> 172.21.15.77:0/3855559865 conn(0x7f00780b1820 msgr2=0x7f00780b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f00803a2640 1 -- 172.21.15.77:0/3855559865 shutdown_connections 2024-04-25T01:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.546+0000 7f00803a2640 1 -- 172.21.15.77:0/3855559865 wait complete. 2024-04-25T01:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.547+0000 7f00803a2640 1 Processor -- start 2024-04-25T01:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f00803a2640 1 -- start start 2024-04-25T01:21:55.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f00803a2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0078143940 0x7f007815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f00803a2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 0x7f00780ce870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f00803a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f007815ecf0 0x7f0078140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f00803a2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0078071fb0 con 0x7f007815ecf0 2024-04-25T01:21:55.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f00803a2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0078071e30 con 0x7f0078143940 2024-04-25T01:21:55.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f00803a2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0078072130 con 0x7f0078143d20 2024-04-25T01:21:55.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f007e93f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f007815ecf0 0x7f0078140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f007e93f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f007815ecf0 0x7f0078140f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40862/0 (socket says 172.21.15.77:40862) 2024-04-25T01:21:55.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f007e93f640 1 -- 172.21.15.77:0/3653909027 learned_addr learned my addr 172.21.15.77:0/3653909027 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f007d93d640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 0x7f00780ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.548+0000 7f007e13e640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0078143940 0x7f007815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f007e93f640 1 -- 172.21.15.77:0/3653909027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 msgr2=0x7f00780ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f007e93f640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 0x7f00780ce870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f007e93f640 1 -- 172.21.15.77:0/3653909027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0078143940 msgr2=0x7f007815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f007e93f640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0078143940 0x7f007815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f007e93f640 1 -- 172.21.15.77:0/3653909027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00780ae7f0 con 0x7f007815ecf0 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f007d93d640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 0x7f00780ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f007e93f640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f007815ecf0 0x7f0078140f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f007000c3c0 tx=0x7f007000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f006effd640 1 -- 172.21.15.77:0/3653909027 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070017070 con 0x7f007815ecf0 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f006effd640 1 -- 172.21.15.77:0/3653909027 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0070005ce0 con 0x7f007815ecf0 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f00803a2640 1 -- 172.21.15.77:0/3653909027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00780aead0 con 0x7f007815ecf0 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f006effd640 1 -- 172.21.15.77:0/3653909027 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070007310 con 0x7f007815ecf0 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.549+0000 7f00803a2640 1 -- 172.21.15.77:0/3653909027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00780af090 con 0x7f007815ecf0 2024-04-25T01:21:55.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.550+0000 7f00803a2640 1 -- 172.21.15.77:0/3653909027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0078068fb0 con 0x7f007815ecf0 2024-04-25T01:21:55.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.550+0000 7f006effd640 1 -- 172.21.15.77:0/3653909027 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f007001a070 con 0x7f007815ecf0 2024-04-25T01:21:55.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.551+0000 7f006effd640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f005c06be30 0x7f005c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.551+0000 7f006effd640 1 -- 172.21.15.77:0/3653909027 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f007008a430 con 0x7f007815ecf0 2024-04-25T01:21:55.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.551+0000 7f007e13e640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f005c06be30 0x7f005c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.551+0000 7f007e13e640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f005c06be30 0x7f005c06e2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f00740113e0 tx=0x7f007400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:55.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.555+0000 7f006effd640 1 -- 172.21.15.77:0/3653909027 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f00700584e0 con 0x7f007815ecf0 2024-04-25T01:21:55.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.729+0000 7f00803a2640 1 -- 172.21.15.77:0/3653909027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f007806d150 con 0x7f007815ecf0 2024-04-25T01:21:55.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.730+0000 7f006effd640 1 -- 172.21.15.77:0/3653909027 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f007005c7e0 con 0x7f007815ecf0 2024-04-25T01:21:55.712 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:55.712 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:21:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 -- 172.21.15.77:0/3653909027 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f005c06be30 msgr2=0x7f005c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f005c06be30 0x7f005c06e2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f00740113e0 tx=0x7f007400b040 comp rx=0 tx=0).stop 2024-04-25T01:21:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 -- 172.21.15.77:0/3653909027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f007815ecf0 msgr2=0x7f0078140f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f007815ecf0 0x7f0078140f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f007000c3c0 tx=0x7f007000c880 comp rx=0 tx=0).stop 2024-04-25T01:21:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 -- 172.21.15.77:0/3653909027 shutdown_connections 2024-04-25T01:21:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f005c06be30 0x7f005c06e2f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f007815ecf0 0x7f0078140f30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0078143d20 0x7f00780ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 --2- 172.21.15.77:0/3653909027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0078143940 0x7f007815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 -- 172.21.15.77:0/3653909027 >> 172.21.15.77:0/3653909027 conn(0x7f00780b1820 msgr2=0x7f00780724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 -- 172.21.15.77:0/3653909027 shutdown_connections 2024-04-25T01:21:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.732+0000 7f00803a2640 1 -- 172.21.15.77:0/3653909027 wait complete. 2024-04-25T01:21:55.724 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:21:55.725 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:21:55.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.842+0000 7f75f1f22640 1 Processor -- start 2024-04-25T01:21:55.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.842+0000 7f75f1f22640 1 -- start start 2024-04-25T01:21:55.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75f1f22640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec158a70 0x7f75ec0b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75f1f22640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f75ec15bc30 0x7f75ec0b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75f1f22640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75ec15c210 con 0x7f75ec0b37c0 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75f1f22640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f75ec0b2ce0 con 0x7f75ec15bc30 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75f1f22640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f75ec0b2fc0 con 0x7f75ec158a70 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75ebfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f75ec15bc30 0x7f75ec0b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75ebfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f75ec15bc30 0x7f75ec0b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40870/0 (socket says 172.21.15.77:40870) 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75ebfff640 1 -- 172.21.15.77:0/3843130002 learned_addr learned my addr 172.21.15.77:0/3843130002 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75eaffd640 1 --1- 172.21.15.77:0/3843130002 >> v1:172.21.15.77:6789/0 conn(0x7f75ec0b37c0 0x7f75ec15b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53900/0 (socket says 172.21.15.77:53900) 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75eb7fe640 1 --2- 172.21.15.77:0/3843130002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec158a70 0x7f75ec0b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75ebfff640 1 -- 172.21.15.77:0/3843130002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec158a70 msgr2=0x7f75ec0b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75ebfff640 1 --2- 172.21.15.77:0/3843130002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec158a70 0x7f75ec0b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75ebfff640 1 -- 172.21.15.77:0/3843130002 >> v1:172.21.15.77:6789/0 conn(0x7f75ec0b37c0 legacy=0x7f75ec15b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.843+0000 7f75ebfff640 1 -- 172.21.15.77:0/3843130002 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec0b32a0 con 0x7f75ec15bc30 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.844+0000 7f75ebfff640 1 --2- 172.21.15.77:0/3843130002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75ec15bc30 0x7f75ec0b4600 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f75e400d150 tx=0x7f75e402c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c308cdab61cd4cf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.844+0000 7f75ea7fc640 1 -- 172.21.15.77:0/3843130002 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4009600 con 0x7f75ec15bc30 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.844+0000 7f75ea7fc640 1 -- 172.21.15.77:0/3843130002 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e4037070 con 0x7f75ec15bc30 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.844+0000 7f75ea7fc640 1 -- 172.21.15.77:0/3843130002 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e40099c0 con 0x7f75ec15bc30 2024-04-25T01:21:55.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.844+0000 7f75f1f22640 1 -- 172.21.15.77:0/3843130002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75ec15bc30 msgr2=0x7f75ec0b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:55.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.844+0000 7f75f1f22640 1 --2- 172.21.15.77:0/3843130002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75ec15bc30 0x7f75ec0b4600 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f75e400d150 tx=0x7f75e402c930 comp rx=0 tx=0).stop 2024-04-25T01:21:55.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.844+0000 7f75f1f22640 1 -- 172.21.15.77:0/3843130002 shutdown_connections 2024-04-25T01:21:55.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.844+0000 7f75f1f22640 1 --2- 172.21.15.77:0/3843130002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75ec15bc30 0x7f75ec0b4600 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.844+0000 7f75f1f22640 1 --2- 172.21.15.77:0/3843130002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec158a70 0x7f75ec0b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.844+0000 7f75f1f22640 1 -- 172.21.15.77:0/3843130002 >> 172.21.15.77:0/3843130002 conn(0x7f75ec06b720 msgr2=0x7f75ec0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:55.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.845+0000 7f75f1f22640 1 -- 172.21.15.77:0/3843130002 shutdown_connections 2024-04-25T01:21:55.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.845+0000 7f75f1f22640 1 -- 172.21.15.77:0/3843130002 wait complete. 2024-04-25T01:21:55.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.846+0000 7f75f1f22640 1 Processor -- start 2024-04-25T01:21:55.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.846+0000 7f75f1f22640 1 -- start start 2024-04-25T01:21:55.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.846+0000 7f75f1f22640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75ec0b37c0 0x7f75ec141710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.846+0000 7f75f1f22640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec158a70 0x7f75ec143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.846+0000 7f75f1f22640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec15bc30 0x7f75ec15c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.846+0000 7f75f1f22640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec06cb70 con 0x7f75ec0b37c0 2024-04-25T01:21:55.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.846+0000 7f75f1f22640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec06c9f0 con 0x7f75ec158a70 2024-04-25T01:21:55.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.846+0000 7f75f1f22640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f75ec06ccf0 con 0x7f75ec15bc30 2024-04-25T01:21:55.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75eb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75ec0b37c0 0x7f75ec141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75eb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75ec0b37c0 0x7f75ec141710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40882/0 (socket says 172.21.15.77:40882) 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75eb7fe640 1 -- 172.21.15.77:0/1411086979 learned_addr learned my addr 172.21.15.77:0/1411086979 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75eaffd640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec158a70 0x7f75ec143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75ebfff640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec15bc30 0x7f75ec15c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75eb7fe640 1 -- 172.21.15.77:0/1411086979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec15bc30 msgr2=0x7f75ec15c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75eb7fe640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec15bc30 0x7f75ec15c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75eb7fe640 1 -- 172.21.15.77:0/1411086979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec158a70 msgr2=0x7f75ec143370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75eb7fe640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec158a70 0x7f75ec143370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75eb7fe640 1 -- 172.21.15.77:0/1411086979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec0ce100 con 0x7f75ec0b37c0 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75ebfff640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec15bc30 0x7f75ec15c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75eb7fe640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75ec0b37c0 0x7f75ec141710 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00d800 tx=0x7f75dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75d3fff640 1 -- 172.21.15.77:0/1411086979 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc00b6c0 con 0x7f75ec0b37c0 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75d3fff640 1 -- 172.21.15.77:0/1411086979 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75dc019070 con 0x7f75ec0b37c0 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75f1f22640 1 -- 172.21.15.77:0/1411086979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ec0ce690 con 0x7f75ec0b37c0 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75d3fff640 1 -- 172.21.15.77:0/1411086979 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc00ba80 con 0x7f75ec0b37c0 2024-04-25T01:21:55.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.847+0000 7f75f1f22640 1 -- 172.21.15.77:0/1411086979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ec141c50 con 0x7f75ec0b37c0 2024-04-25T01:21:55.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.848+0000 7f75f1f22640 1 -- 172.21.15.77:0/1411086979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75ec068fb0 con 0x7f75ec0b37c0 2024-04-25T01:21:55.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.849+0000 7f75d3fff640 1 -- 172.21.15.77:0/1411086979 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f75dc00bc20 con 0x7f75ec0b37c0 2024-04-25T01:21:55.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.849+0000 7f75d3fff640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75d806be30 0x7f75d806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:55.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.849+0000 7f75d3fff640 1 -- 172.21.15.77:0/1411086979 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f75dc08b5d0 con 0x7f75ec0b37c0 2024-04-25T01:21:55.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.849+0000 7f75eaffd640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75d806be30 0x7f75d806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:55.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.850+0000 7f75eaffd640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75d806be30 0x7f75d806e2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0af5b0 tx=0x7f75e0007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:55.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:55.853+0000 7f75d3fff640 1 -- 172.21.15.77:0/1411086979 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f75dc059c40 con 0x7f75ec0b37c0 2024-04-25T01:21:56.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.026+0000 7f75f1f22640 1 -- 172.21.15.77:0/1411086979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f75ec0b3c70 con 0x7f75ec0b37c0 2024-04-25T01:21:56.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.026+0000 7f75d3fff640 1 -- 172.21.15.77:0/1411086979 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f75dc059540 con 0x7f75ec0b37c0 2024-04-25T01:21:56.008 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:21:56.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 -- 172.21.15.77:0/1411086979 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75d806be30 msgr2=0x7f75d806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75d806be30 0x7f75d806e2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0af5b0 tx=0x7f75e0007e80 comp rx=0 tx=0).stop 2024-04-25T01:21:56.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 -- 172.21.15.77:0/1411086979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75ec0b37c0 msgr2=0x7f75ec141710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75ec0b37c0 0x7f75ec141710 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00d800 tx=0x7f75dc00dcc0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 -- 172.21.15.77:0/1411086979 shutdown_connections 2024-04-25T01:21:56.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75d806be30 0x7f75d806e2f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75ec15bc30 0x7f75ec15c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75ec158a70 0x7f75ec143370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 --2- 172.21.15.77:0/1411086979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75ec0b37c0 0x7f75ec141710 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 -- 172.21.15.77:0/1411086979 >> 172.21.15.77:0/1411086979 conn(0x7f75ec06b720 msgr2=0x7f75ec159710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 -- 172.21.15.77:0/1411086979 shutdown_connections 2024-04-25T01:21:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.028+0000 7f75f1f22640 1 -- 172.21.15.77:0/1411086979 wait complete. 2024-04-25T01:21:56.021 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:21:56.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.139+0000 7f86a60fc640 1 Processor -- start 2024-04-25T01:21:56.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.139+0000 7f86a60fc640 1 -- start start 2024-04-25T01:21:56.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.140+0000 7f86a60fc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f86a015e460 0x7f86a00d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.140+0000 7f86a60fc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a00d0de0 con 0x7f86a00cebe0 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.140+0000 7f86a60fc640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a0070280 con 0x7f86a00d09a0 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.140+0000 7f86a60fc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f86a006c230 con 0x7f86a015e460 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.140+0000 7f869ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f86a00cebe0 0x7f86a00ccf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53912/0 (socket says 172.21.15.77:53912) 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.140+0000 7f869ffff640 1 -- 172.21.15.77:0/4142264245 learned_addr learned my addr 172.21.15.77:0/4142264245 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.140+0000 7f869f7fe640 1 --2- 172.21.15.77:0/4142264245 >> v2:172.21.15.121:3300/0 conn(0x7f86a015e460 0x7f86a00d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.140+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2779720488 0 0) 0x7f86a00d0de0 con 0x7f86a00cebe0 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.140+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f868c003520 con 0x7f86a00cebe0 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3570043330 0 0) 0x7f868c003520 con 0x7f86a00cebe0 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 >> v2:172.21.15.121:3300/0 conn(0x7f86a015e460 msgr2=0x7f86a00d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 --2- 172.21.15.77:0/4142264245 >> v2:172.21.15.121:3300/0 conn(0x7f86a015e460 0x7f86a00d2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 >> v1:172.21.15.121:6789/0 conn(0x7f86a00d09a0 legacy=0x7f86a00ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a00d1080 con 0x7f86a00cebe0 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f86980019f0 con 0x7f86a00cebe0 2024-04-25T01:21:56.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 >> v1:172.21.15.77:6789/0 conn(0x7f86a00cebe0 legacy=0x7f86a00ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 --2- 172.21.15.77:0/4142264245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f868c0039b0 0x7f868c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 --2- 172.21.15.77:0/4142264245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f868c004800 0x7f868c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 --2- 172.21.15.77:0/4142264245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 0x7f868c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f868c00c040 con 0x7f868c0051c0 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f868c00c1f0 con 0x7f868c004800 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f868c00c3f0 con 0x7f868c0039b0 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869ffff640 1 --2- 172.21.15.77:0/4142264245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 0x7f868c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f8697fff640 1 --2- 172.21.15.77:0/4142264245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f868c004800 0x7f868c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869f7fe640 1 --2- 172.21.15.77:0/4142264245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f868c0039b0 0x7f868c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869ffff640 1 -- 172.21.15.77:0/4142264245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f868c0039b0 msgr2=0x7f868c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869ffff640 1 --2- 172.21.15.77:0/4142264245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f868c0039b0 0x7f868c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869ffff640 1 -- 172.21.15.77:0/4142264245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f868c004800 msgr2=0x7f868c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869ffff640 1 --2- 172.21.15.77:0/4142264245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f868c004800 0x7f868c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.141+0000 7f869ffff640 1 -- 172.21.15.77:0/4142264245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868c003520 con 0x7f868c0051c0 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.142+0000 7f869ffff640 1 -- 172.21.15.77:0/4142264245 reap_dead start 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.142+0000 7f869ffff640 1 --2- 172.21.15.77:0/4142264245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 0x7f868c00b9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f86a0072650 tx=0x7f868c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85171aa8cb544244 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.142+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869800a4a0 con 0x7f868c0051c0 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.142+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f869802f070 con 0x7f868c0051c0 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.142+0000 7f869effd640 1 -- 172.21.15.77:0/4142264245 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869802bcb0 con 0x7f868c0051c0 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.142+0000 7f86a60fc640 1 -- 172.21.15.77:0/4142264245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 msgr2=0x7f868c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.142+0000 7f86a60fc640 1 --2- 172.21.15.77:0/4142264245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 0x7f868c00b9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f86a0072650 tx=0x7f868c005830 comp rx=0 tx=0).stop 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.142+0000 7f86a60fc640 1 -- 172.21.15.77:0/4142264245 shutdown_connections 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.142+0000 7f86a60fc640 1 --2- 172.21.15.77:0/4142264245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 0x7f868c00b9c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.142+0000 7f86a60fc640 1 -- 172.21.15.77:0/4142264245 >> 172.21.15.77:0/4142264245 conn(0x7f86a00d1c50 msgr2=0x7f86a00d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.143+0000 7f86a60fc640 1 -- 172.21.15.77:0/4142264245 shutdown_connections 2024-04-25T01:21:56.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.143+0000 7f86a60fc640 1 -- 172.21.15.77:0/4142264245 wait complete. 2024-04-25T01:21:56.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.144+0000 7f86a60fc640 1 Processor -- start 2024-04-25T01:21:56.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.144+0000 7f86a60fc640 1 -- start start 2024-04-25T01:21:56.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.144+0000 7f86a60fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 0x7f86a014d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.144+0000 7f86a60fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86a0154cf0 0x7f86a0155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.144+0000 7f86a60fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86a0151640 0x7f86a0151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.144+0000 7f86a60fc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86a00cc720 con 0x7f868c0051c0 2024-04-25T01:21:56.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.144+0000 7f86a60fc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f86a00cca90 con 0x7f86a0151640 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.144+0000 7f86a60fc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f86a00d2770 con 0x7f86a0154cf0 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.144+0000 7f869f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 0x7f86a014d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.144+0000 7f869f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 0x7f86a014d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40906/0 (socket says 172.21.15.77:40906) 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869f7fe640 1 -- 172.21.15.77:0/2514604136 learned_addr learned my addr 172.21.15.77:0/2514604136 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869ffff640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86a0151640 0x7f86a0151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869f7fe640 1 -- 172.21.15.77:0/2514604136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86a0154cf0 msgr2=0x7f86a0155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f8697fff640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86a0154cf0 0x7f86a0155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869f7fe640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86a0154cf0 0x7f86a0155170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869f7fe640 1 -- 172.21.15.77:0/2514604136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86a0151640 msgr2=0x7f86a0151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869f7fe640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86a0151640 0x7f86a0151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869f7fe640 1 -- 172.21.15.77:0/2514604136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a0152000 con 0x7f868c0051c0 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869ffff640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86a0151640 0x7f86a0151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869f7fe640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 0x7f86a014d7c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8690011910 tx=0x7f8690011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:56.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869cff9640 1 -- 172.21.15.77:0/2514604136 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869000f9d0 con 0x7f868c0051c0 2024-04-25T01:21:56.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869cff9640 1 -- 172.21.15.77:0/2514604136 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f869000fb70 con 0x7f868c0051c0 2024-04-25T01:21:56.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f86a60fc640 1 -- 172.21.15.77:0/2514604136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a0152290 con 0x7f868c0051c0 2024-04-25T01:21:56.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f869cff9640 1 -- 172.21.15.77:0/2514604136 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690014440 con 0x7f868c0051c0 2024-04-25T01:21:56.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.145+0000 7f86a60fc640 1 -- 172.21.15.77:0/2514604136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a0152850 con 0x7f868c0051c0 2024-04-25T01:21:56.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.146+0000 7f86a60fc640 1 -- 172.21.15.77:0/2514604136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a0068fb0 con 0x7f868c0051c0 2024-04-25T01:21:56.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.147+0000 7f869cff9640 1 -- 172.21.15.77:0/2514604136 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8690012040 con 0x7f868c0051c0 2024-04-25T01:21:56.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.147+0000 7f869cff9640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f868006be60 0x7f868006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.147+0000 7f869cff9640 1 -- 172.21.15.77:0/2514604136 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f869008b670 con 0x7f868c0051c0 2024-04-25T01:21:56.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.147+0000 7f8697fff640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f868006be60 0x7f868006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.147+0000 7f8697fff640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f868006be60 0x7f868006e320 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8688000c00 tx=0x7f8688009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:56.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.151+0000 7f869cff9640 1 -- 172.21.15.77:0/2514604136 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8690019030 con 0x7f868c0051c0 2024-04-25T01:21:56.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.324+0000 7f86a60fc640 1 -- 172.21.15.77:0/2514604136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f86a00cf260 con 0x7f868c0051c0 2024-04-25T01:21:56.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.325+0000 7f869cff9640 1 -- 172.21.15.77:0/2514604136 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f8690059ce0 con 0x7f868c0051c0 2024-04-25T01:21:56.307 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:21:56.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.326+0000 7f86a60fc640 1 -- 172.21.15.77:0/2514604136 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f868006be60 msgr2=0x7f868006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.326+0000 7f86a60fc640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f868006be60 0x7f868006e320 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8688000c00 tx=0x7f8688009040 comp rx=0 tx=0).stop 2024-04-25T01:21:56.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.327+0000 7f86a60fc640 1 -- 172.21.15.77:0/2514604136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 msgr2=0x7f86a014d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.327+0000 7f86a60fc640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 0x7f86a014d7c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8690011910 tx=0x7f8690011dd0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.327+0000 7f86a60fc640 1 -- 172.21.15.77:0/2514604136 shutdown_connections 2024-04-25T01:21:56.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.327+0000 7f86a60fc640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f868006be60 0x7f868006e320 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.327+0000 7f86a60fc640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86a0151640 0x7f86a0151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.327+0000 7f86a60fc640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86a0154cf0 0x7f86a0155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.327+0000 7f86a60fc640 1 --2- 172.21.15.77:0/2514604136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c0051c0 0x7f86a014d7c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.327+0000 7f86a60fc640 1 -- 172.21.15.77:0/2514604136 >> 172.21.15.77:0/2514604136 conn(0x7f86a00d1c50 msgr2=0x7f86a00d2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:56.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.327+0000 7f86a60fc640 1 -- 172.21.15.77:0/2514604136 shutdown_connections 2024-04-25T01:21:56.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.327+0000 7f86a60fc640 1 -- 172.21.15.77:0/2514604136 wait complete. 2024-04-25T01:21:56.319 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:21:56.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.436+0000 7f9f8b6f5640 1 Processor -- start 2024-04-25T01:21:56.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.437+0000 7f9f8b6f5640 1 -- start start 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.437+0000 7f9f8b6f5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9f840b4ac0 0x7f9f84160300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.437+0000 7f9f8b6f5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f84160840 con 0x7f9f8415f6a0 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.437+0000 7f9f8b6f5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f840cd380 con 0x7f9f840b2be0 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.437+0000 7f9f8b6f5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9f84160b00 con 0x7f9f840b4ac0 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.437+0000 7f9f89491640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9f8415f6a0 0x7f9f8415fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53914/0 (socket says 172.21.15.77:53914) 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.437+0000 7f9f89491640 1 -- 172.21.15.77:0/3887443 learned_addr learned my addr 172.21.15.77:0/3887443 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f88c90640 1 --2- 172.21.15.77:0/3887443 >> v2:172.21.15.121:3300/0 conn(0x7f9f840b4ac0 0x7f9f84160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3920187171 0 0) 0x7f9f84160840 con 0x7f9f8415f6a0 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f70003520 con 0x7f9f8415f6a0 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3455587595 0 0) 0x7f9f70003520 con 0x7f9f8415f6a0 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 >> v2:172.21.15.121:3300/0 conn(0x7f9f840b4ac0 msgr2=0x7f9f84160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:56.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 --2- 172.21.15.77:0/3887443 >> v2:172.21.15.121:3300/0 conn(0x7f9f840b4ac0 0x7f9f84160300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 >> v1:172.21.15.121:6789/0 conn(0x7f9f840b2be0 legacy=0x7f9f840cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f840cd660 con 0x7f9f8415f6a0 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f9f800019f0 con 0x7f9f8415f6a0 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 >> v1:172.21.15.77:6789/0 conn(0x7f9f8415f6a0 legacy=0x7f9f8415fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 --2- 172.21.15.77:0/3887443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700039b0 0x7f9f70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 --2- 172.21.15.77:0/3887443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f70004800 0x7f9f70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 --2- 172.21.15.77:0/3887443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f700052e0 0x7f9f7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9f84160b00 con 0x7f9f700052e0 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7000c1b0 con 0x7f9f70004800 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9f7000c3b0 con 0x7f9f700039b0 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f89c92640 1 --2- 172.21.15.77:0/3887443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f700052e0 0x7f9f7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f88c90640 1 --2- 172.21.15.77:0/3887443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f70004800 0x7f9f70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f89491640 1 --2- 172.21.15.77:0/3887443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700039b0 0x7f9f70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f89c92640 1 -- 172.21.15.77:0/3887443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700039b0 msgr2=0x7f9f70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:56.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f89c92640 1 --2- 172.21.15.77:0/3887443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700039b0 0x7f9f70003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f89c92640 1 -- 172.21.15.77:0/3887443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f70004800 msgr2=0x7f9f70004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f89c92640 1 --2- 172.21.15.77:0/3887443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f70004800 0x7f9f70004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.438+0000 7f9f89c92640 1 -- 172.21.15.77:0/3887443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f70003520 con 0x7f9f700052e0 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.439+0000 7f9f89c92640 1 -- 172.21.15.77:0/3887443 reap_dead start 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.439+0000 7f9f89c92640 1 --2- 172.21.15.77:0/3887443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f700052e0 0x7f9f7000b9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c001720 tx=0x7f9f70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b786f7841daee199 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.439+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c001370 con 0x7f9f700052e0 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.439+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f7c0052d0 con 0x7f9f700052e0 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.439+0000 7f9f7bfff640 1 -- 172.21.15.77:0/3887443 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c02abc0 con 0x7f9f700052e0 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.440+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/3887443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f700052e0 msgr2=0x7f9f7000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.440+0000 7f9f8b6f5640 1 --2- 172.21.15.77:0/3887443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f700052e0 0x7f9f7000b9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c001720 tx=0x7f9f70005830 comp rx=0 tx=0).stop 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.440+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/3887443 shutdown_connections 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.440+0000 7f9f8b6f5640 1 --2- 172.21.15.77:0/3887443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f700052e0 0x7f9f7000b9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.440+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/3887443 >> 172.21.15.77:0/3887443 conn(0x7f9f840b37a0 msgr2=0x7f9f840b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:56.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.440+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/3887443 shutdown_connections 2024-04-25T01:21:56.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.440+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/3887443 wait complete. 2024-04-25T01:21:56.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.441+0000 7f9f8b6f5640 1 Processor -- start 2024-04-25T01:21:56.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.441+0000 7f9f8b6f5640 1 -- start start 2024-04-25T01:21:56.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f8b6f5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f700052e0 0x7f9f84152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f8b6f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f84153390 0x7f9f840c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f8b6f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f840c7540 0x7f9f841f50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f8b6f5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9f840b42c0 con 0x7f9f84153390 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f8b6f5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9f840cf840 con 0x7f9f700052e0 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f8b6f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9f840b4440 con 0x7f9f840c7540 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f88c90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f84153390 0x7f9f840c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f88c90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f84153390 0x7f9f840c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40912/0 (socket says 172.21.15.77:40912) 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f88c90640 1 -- 172.21.15.77:0/1857411239 learned_addr learned my addr 172.21.15.77:0/1857411239 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f89c92640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f840c7540 0x7f9f841f50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f89491640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f700052e0 0x7f9f84152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f88c90640 1 -- 172.21.15.77:0/1857411239 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f840c7540 msgr2=0x7f9f841f50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f88c90640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f840c7540 0x7f9f841f50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f88c90640 1 -- 172.21.15.77:0/1857411239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f700052e0 msgr2=0x7f9f84152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f88c90640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f700052e0 0x7f9f84152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f88c90640 1 -- 172.21.15.77:0/1857411239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f840c5810 con 0x7f9f84153390 2024-04-25T01:21:56.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f89c92640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f840c7540 0x7f9f841f50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f89491640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f700052e0 0x7f9f84152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f88c90640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f84153390 0x7f9f840c7000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9f740118e0 tx=0x7f9f74011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f79ffb640 1 -- 172.21.15.77:0/1857411239 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7400faf0 con 0x7f9f84153390 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f79ffb640 1 -- 172.21.15.77:0/1857411239 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f7400fc90 con 0x7f9f84153390 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f79ffb640 1 -- 172.21.15.77:0/1857411239 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f74014440 con 0x7f9f84153390 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.442+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/1857411239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f84152890 con 0x7f9f84153390 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.443+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/1857411239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f840c5d60 con 0x7f9f84153390 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.444+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/1857411239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f84068180 con 0x7f9f84153390 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.444+0000 7f9f79ffb640 1 -- 172.21.15.77:0/1857411239 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9f74012040 con 0x7f9f84153390 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.444+0000 7f9f79ffb640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9f6806be60 0x7f9f6806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.444+0000 7f9f89491640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9f6806be60 0x7f9f6806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.444+0000 7f9f79ffb640 1 -- 172.21.15.77:0/1857411239 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9f7408b8c0 con 0x7f9f84153390 2024-04-25T01:21:56.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.445+0000 7f9f89491640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9f6806be60 0x7f9f6806e320 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9f80001460 tx=0x7f9f80006040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:56.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.449+0000 7f9f79ffb640 1 -- 172.21.15.77:0/1857411239 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9f74019030 con 0x7f9f84153390 2024-04-25T01:21:56.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.620+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/1857411239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9f8415ea50 con 0x7f9f84153390 2024-04-25T01:21:56.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.620+0000 7f9f79ffb640 1 -- 172.21.15.77:0/1857411239 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f9f74059f30 con 0x7f9f84153390 2024-04-25T01:21:56.602 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/1857411239 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9f6806be60 msgr2=0x7f9f6806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9f6806be60 0x7f9f6806e320 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9f80001460 tx=0x7f9f80006040 comp rx=0 tx=0).stop 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/1857411239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f84153390 msgr2=0x7f9f840c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f84153390 0x7f9f840c7000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9f740118e0 tx=0x7f9f74011da0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/1857411239 shutdown_connections 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9f6806be60 0x7f9f6806e320 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f84153390 0x7f9f840c7000 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f840c7540 0x7f9f841f50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 --2- 172.21.15.77:0/1857411239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f700052e0 0x7f9f84152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/1857411239 >> 172.21.15.77:0/1857411239 conn(0x7f9f840b37a0 msgr2=0x7f9f840d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/1857411239 shutdown_connections 2024-04-25T01:21:56.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.622+0000 7f9f8b6f5640 1 -- 172.21.15.77:0/1857411239 wait complete. 2024-04-25T01:21:56.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:21:56.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.731+0000 7f20046b2640 1 Processor -- start 2024-04-25T01:21:56.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.731+0000 7f20046b2640 1 -- start start 2024-04-25T01:21:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.732+0000 7f20046b2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ffc0d27b0 0x7f1ffc0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.732+0000 7f20046b2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ffc0cea40 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.732+0000 7f20046b2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ffc0cece0 con 0x7f1ffc0b9a90 2024-04-25T01:21:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.732+0000 7f20046b2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f1ffc0cefc0 con 0x7f1ffc0d27b0 2024-04-25T01:21:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.732+0000 7f200244e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1ffc0d23d0 0x7f1ffc0b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47444/0 (socket says 172.21.15.77:47444) 2024-04-25T01:21:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.732+0000 7f200244e640 1 -- 172.21.15.77:0/2282281520 learned_addr learned my addr 172.21.15.77:0/2282281520 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.732+0000 7f2002c4f640 1 --2- 172.21.15.77:0/2282281520 >> v2:172.21.15.77:3300/0 conn(0x7f1ffc0d27b0 0x7f1ffc0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.733+0000 7f2002c4f640 1 -- 172.21.15.77:0/2282281520 >> v1:172.21.15.121:6789/0 conn(0x7f1ffc0b9a90 legacy=0x7f1ffc0b6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.733+0000 7f200144c640 1 -- 172.21.15.77:0/2282281520 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 488088851 0 0) 0x7f1ffc0cea40 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.733+0000 7f2002c4f640 1 -- 172.21.15.77:0/2282281520 >> v1:172.21.15.77:6789/0 conn(0x7f1ffc0d23d0 legacy=0x7f1ffc0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.733+0000 7f2002c4f640 1 -- 172.21.15.77:0/2282281520 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc0cf2a0 con 0x7f1ffc0d27b0 2024-04-25T01:21:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.733+0000 7f2002c4f640 1 --2- 172.21.15.77:0/2282281520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ffc0d27b0 0x7f1ffc0d4920 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40097a0 tx=0x7f1ff402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55066b6edcd638e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.734+0000 7f200144c640 1 -- 172.21.15.77:0/2282281520 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4037070 con 0x7f1ffc0d27b0 2024-04-25T01:21:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.734+0000 7f200144c640 1 -- 172.21.15.77:0/2282281520 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff4032440 con 0x7f1ffc0d27b0 2024-04-25T01:21:56.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.734+0000 7f200144c640 1 -- 172.21.15.77:0/2282281520 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4032810 con 0x7f1ffc0d27b0 2024-04-25T01:21:56.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.734+0000 7f20046b2640 1 -- 172.21.15.77:0/2282281520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ffc0d27b0 msgr2=0x7f1ffc0d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.734+0000 7f20046b2640 1 --2- 172.21.15.77:0/2282281520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ffc0d27b0 0x7f1ffc0d4920 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40097a0 tx=0x7f1ff402a270 comp rx=0 tx=0).stop 2024-04-25T01:21:56.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.734+0000 7f20046b2640 1 -- 172.21.15.77:0/2282281520 shutdown_connections 2024-04-25T01:21:56.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.734+0000 7f20046b2640 1 --2- 172.21.15.77:0/2282281520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ffc0d27b0 0x7f1ffc0d4920 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.734+0000 7f20046b2640 1 -- 172.21.15.77:0/2282281520 >> 172.21.15.77:0/2282281520 conn(0x7f1ffc0d3e30 msgr2=0x7f1ffc0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:56.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.734+0000 7f20046b2640 1 -- 172.21.15.77:0/2282281520 shutdown_connections 2024-04-25T01:21:56.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.734+0000 7f20046b2640 1 -- 172.21.15.77:0/2282281520 wait complete. 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.735+0000 7f20046b2640 1 Processor -- start 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.735+0000 7f20046b2640 1 -- start start 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.736+0000 7f20046b2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ffc0b9a90 0x7f1ffc153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.736+0000 7f20046b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ffc0d23d0 0x7f1ffc0c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.736+0000 7f20046b2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ffc0d27b0 0x7f1ffc1520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.736+0000 7f20046b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc0cfd80 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.736+0000 7f20046b2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc0cf860 con 0x7f1ffc0d27b0 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.736+0000 7f20046b2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1ffc0d0ee0 con 0x7f1ffc0b9a90 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.736+0000 7f2001c4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ffc0d23d0 0x7f1ffc0c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.736+0000 7f2001c4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ffc0d23d0 0x7f1ffc0c9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60928/0 (socket says 172.21.15.77:60928) 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.736+0000 7f2001c4d640 1 -- 172.21.15.77:0/3657845294 learned_addr learned my addr 172.21.15.77:0/3657845294 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:56.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.736+0000 7f2002c4f640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ffc0d27b0 0x7f1ffc1520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.736+0000 7f200244e640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ffc0b9a90 0x7f1ffc153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f2001c4d640 1 -- 172.21.15.77:0/3657845294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ffc0b9a90 msgr2=0x7f1ffc153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f2001c4d640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ffc0b9a90 0x7f1ffc153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f2001c4d640 1 -- 172.21.15.77:0/3657845294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ffc0d27b0 msgr2=0x7f1ffc1520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f2001c4d640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ffc0d27b0 0x7f1ffc1520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f2001c4d640 1 -- 172.21.15.77:0/3657845294 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc1543c0 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f2002c4f640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ffc0d27b0 0x7f1ffc1520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f2001c4d640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ffc0d23d0 0x7f1ffc0c9ba0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c430 tx=0x7f1fec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f1ff2ffd640 1 -- 172.21.15.77:0/3657845294 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec011790 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f1ff2ffd640 1 -- 172.21.15.77:0/3657845294 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fec017070 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f20046b2640 1 -- 172.21.15.77:0/3657845294 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ffc0cb150 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f1ff2ffd640 1 -- 172.21.15.77:0/3657845294 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec011bb0 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.737+0000 7f20046b2640 1 -- 172.21.15.77:0/3657845294 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ffc0cb710 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.738+0000 7f20046b2640 1 -- 172.21.15.77:0/3657845294 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ffc068fb0 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.738+0000 7f1ff2ffd640 1 -- 172.21.15.77:0/3657845294 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1fec01c460 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.739+0000 7f1ff2ffd640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fe006be30 0x7f1fe006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:56.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.739+0000 7f1ff2ffd640 1 -- 172.21.15.77:0/3657845294 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1fec08aff0 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.739+0000 7f200244e640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fe006be30 0x7f1fe006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:56.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.739+0000 7f200244e640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fe006be30 0x7f1fe006e2f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80012e0 tx=0x7f1ff8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.743+0000 7f1ff2ffd640 1 -- 172.21.15.77:0/3657845294 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1fec0596f0 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.913+0000 7f20046b2640 1 -- 172.21.15.77:0/3657845294 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1ffc0b6b80 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.913+0000 7f1ff2ffd640 1 -- 172.21.15.77:0/3657845294 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f1fec058ff0 con 0x7f1ffc0d23d0 2024-04-25T01:21:56.895 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:21:56.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 -- 172.21.15.77:0/3657845294 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fe006be30 msgr2=0x7f1fe006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fe006be30 0x7f1fe006e2f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80012e0 tx=0x7f1ff8009040 comp rx=0 tx=0).stop 2024-04-25T01:21:56.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 -- 172.21.15.77:0/3657845294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ffc0d23d0 msgr2=0x7f1ffc0c9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:56.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ffc0d23d0 0x7f1ffc0c9ba0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c430 tx=0x7f1fec00c8f0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 -- 172.21.15.77:0/3657845294 shutdown_connections 2024-04-25T01:21:56.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fe006be30 0x7f1fe006e2f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ffc0d23d0 0x7f1ffc0c9ba0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ffc0d27b0 0x7f1ffc1520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 --2- 172.21.15.77:0/3657845294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ffc0b9a90 0x7f1ffc153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:56.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 -- 172.21.15.77:0/3657845294 >> 172.21.15.77:0/3657845294 conn(0x7f1ffc0d3e30 msgr2=0x7f1ffc0b8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:56.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 -- 172.21.15.77:0/3657845294 shutdown_connections 2024-04-25T01:21:56.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:56.915+0000 7f20046b2640 1 -- 172.21.15.77:0/3657845294 wait complete. 2024-04-25T01:21:56.908 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:21:57.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.024+0000 7f13294a5640 1 Processor -- start 2024-04-25T01:21:57.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.024+0000 7f13294a5640 1 -- start start 2024-04-25T01:21:57.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.024+0000 7f13294a5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f132415e380 0x7f13240cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.024+0000 7f13294a5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13240d1070 0x7f13240b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.024+0000 7f13294a5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13240d1eb0 con 0x7f13240cfcf0 2024-04-25T01:21:57.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.024+0000 7f13294a5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f13240d2190 con 0x7f13240d1070 2024-04-25T01:21:57.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.024+0000 7f13294a5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f13240d2470 con 0x7f132415e380 2024-04-25T01:21:57.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.025+0000 7f13237fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13240d1070 0x7f13240b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.025+0000 7f13237fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13240d1070 0x7f13240b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60936/0 (socket says 172.21.15.77:60936) 2024-04-25T01:21:57.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.025+0000 7f13237fe640 1 -- 172.21.15.77:0/2880124310 learned_addr learned my addr 172.21.15.77:0/2880124310 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:57.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.025+0000 7f13237fe640 1 -- 172.21.15.77:0/2880124310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f132415e380 msgr2=0x7f13240cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:57.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.025+0000 7f13237fe640 1 --2- 172.21.15.77:0/2880124310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f132415e380 0x7f13240cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.025+0000 7f13237fe640 1 -- 172.21.15.77:0/2880124310 >> v1:172.21.15.121:6789/0 conn(0x7f13240cfcf0 legacy=0x7f13240d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:57.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.025+0000 7f13237fe640 1 -- 172.21.15.77:0/2880124310 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13240d2750 con 0x7f13240d1070 2024-04-25T01:21:57.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13237fe640 1 --2- 172.21.15.77:0/2880124310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13240d1070 0x7f13240b37d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f131c009e50 tx=0x7f131c02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ca1504ed2db179 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13227fc640 1 -- 172.21.15.77:0/2880124310 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c036070 con 0x7f13240d1070 2024-04-25T01:21:57.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13227fc640 1 -- 172.21.15.77:0/2880124310 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131c031600 con 0x7f13240d1070 2024-04-25T01:21:57.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13227fc640 1 -- 172.21.15.77:0/2880124310 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c0319d0 con 0x7f13240d1070 2024-04-25T01:21:57.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13294a5640 1 -- 172.21.15.77:0/2880124310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13240d1070 msgr2=0x7f13240b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13294a5640 1 --2- 172.21.15.77:0/2880124310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13240d1070 0x7f13240b37d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f131c009e50 tx=0x7f131c02a6c0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13294a5640 1 -- 172.21.15.77:0/2880124310 shutdown_connections 2024-04-25T01:21:57.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13294a5640 1 --2- 172.21.15.77:0/2880124310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13240d1070 0x7f13240b37d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13294a5640 1 --2- 172.21.15.77:0/2880124310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f132415e380 0x7f13240cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13294a5640 1 -- 172.21.15.77:0/2880124310 >> 172.21.15.77:0/2880124310 conn(0x7f132409d200 msgr2=0x7f13240d3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:57.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13294a5640 1 -- 172.21.15.77:0/2880124310 shutdown_connections 2024-04-25T01:21:57.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.026+0000 7f13294a5640 1 -- 172.21.15.77:0/2880124310 wait complete. 2024-04-25T01:21:57.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.027+0000 7f13294a5640 1 Processor -- start 2024-04-25T01:21:57.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.027+0000 7f13294a5640 1 -- start start 2024-04-25T01:21:57.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.028+0000 7f13294a5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13240cfcf0 0x7f1324154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.028+0000 7f13294a5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13240d1070 0x7f13240c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.028+0000 7f13294a5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f132415e380 0x7f1324153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.028+0000 7f13294a5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f13240d2f50 con 0x7f13240d1070 2024-04-25T01:21:57.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.028+0000 7f13294a5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f13240d1b40 con 0x7f132415e380 2024-04-25T01:21:57.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.028+0000 7f13294a5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f13240d30d0 con 0x7f13240cfcf0 2024-04-25T01:21:57.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.028+0000 7f131bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13240d1070 0x7f13240c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.028+0000 7f131bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13240d1070 0x7f13240c8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60952/0 (socket says 172.21.15.77:60952) 2024-04-25T01:21:57.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.028+0000 7f131bfff640 1 -- 172.21.15.77:0/1376157453 learned_addr learned my addr 172.21.15.77:0/1376157453 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:57.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.028+0000 7f13237fe640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f132415e380 0x7f1324153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.028+0000 7f1322ffd640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13240cfcf0 0x7f1324154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f131bfff640 1 -- 172.21.15.77:0/1376157453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13240cfcf0 msgr2=0x7f1324154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f131bfff640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13240cfcf0 0x7f1324154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f131bfff640 1 -- 172.21.15.77:0/1376157453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f132415e380 msgr2=0x7f1324153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f131bfff640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f132415e380 0x7f1324153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f131bfff640 1 -- 172.21.15.77:0/1376157453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13240ca010 con 0x7f13240d1070 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f1322ffd640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13240cfcf0 0x7f1324154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f13237fe640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f132415e380 0x7f1324153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f131bfff640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13240d1070 0x7f13240c8af0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f130c00c430 tx=0x7f130c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f131b7fe640 1 -- 172.21.15.77:0/1376157453 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c011790 con 0x7f13240d1070 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f131b7fe640 1 -- 172.21.15.77:0/1376157453 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f130c017070 con 0x7f13240d1070 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f131b7fe640 1 -- 172.21.15.77:0/1376157453 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c011bb0 con 0x7f13240d1070 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f13294a5640 1 -- 172.21.15.77:0/1376157453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13240ca2f0 con 0x7f13240d1070 2024-04-25T01:21:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.029+0000 7f13294a5640 1 -- 172.21.15.77:0/1376157453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13240c82f0 con 0x7f13240d1070 2024-04-25T01:21:57.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.030+0000 7f13294a5640 1 -- 172.21.15.77:0/1376157453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1324068180 con 0x7f13240d1070 2024-04-25T01:21:57.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.030+0000 7f131b7fe640 1 -- 172.21.15.77:0/1376157453 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f130c01c460 con 0x7f13240d1070 2024-04-25T01:21:57.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.031+0000 7f131b7fe640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f130406be30 0x7f130406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.031+0000 7f131b7fe640 1 -- 172.21.15.77:0/1376157453 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f130c08b090 con 0x7f13240d1070 2024-04-25T01:21:57.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.031+0000 7f1322ffd640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f130406be30 0x7f130406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.031+0000 7f1322ffd640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f130406be30 0x7f130406e2f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f131400a870 tx=0x7f1314008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.035+0000 7f131b7fe640 1 -- 172.21.15.77:0/1376157453 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f130c059700 con 0x7f13240d1070 2024-04-25T01:21:57.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.220+0000 7f13294a5640 1 -- 172.21.15.77:0/1376157453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f13240d0530 con 0x7f13240d1070 2024-04-25T01:21:57.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.221+0000 7f131b7fe640 1 -- 172.21.15.77:0/1376157453 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f130c059000 con 0x7f13240d1070 2024-04-25T01:21:57.203 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-04-25T01:21:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 -- 172.21.15.77:0/1376157453 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f130406be30 msgr2=0x7f130406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f130406be30 0x7f130406e2f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f131400a870 tx=0x7f1314008040 comp rx=0 tx=0).stop 2024-04-25T01:21:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 -- 172.21.15.77:0/1376157453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13240d1070 msgr2=0x7f13240c8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13240d1070 0x7f13240c8af0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f130c00c430 tx=0x7f130c00c8f0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 -- 172.21.15.77:0/1376157453 shutdown_connections 2024-04-25T01:21:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f130406be30 0x7f130406e2f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13240d1070 0x7f13240c8af0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f132415e380 0x7f1324153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 --2- 172.21.15.77:0/1376157453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13240cfcf0 0x7f1324154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 -- 172.21.15.77:0/1376157453 >> 172.21.15.77:0/1376157453 conn(0x7f132409d200 msgr2=0x7f1324161e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 -- 172.21.15.77:0/1376157453 shutdown_connections 2024-04-25T01:21:57.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.223+0000 7f13294a5640 1 -- 172.21.15.77:0/1376157453 wait complete. 2024-04-25T01:21:57.215 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:21:57.215 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:21:57.231 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:21:57.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.385+0000 7f91e9810640 1 Processor -- start 2024-04-25T01:21:57.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.385+0000 7f91e9810640 1 -- start start 2024-04-25T01:21:57.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.385+0000 7f91e9810640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f91e415c090 0x7f91e40b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e9810640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f91e40d27a0 0x7f91e40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e9810640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91e40cf9f0 con 0x7f91e40b0d80 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e9810640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f91e40d2c30 con 0x7f91e415c090 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e9810640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f91e40cf0c0 con 0x7f91e40d27a0 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e2ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f91e415c090 0x7f91e40b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e2ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f91e415c090 0x7f91e40b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60964/0 (socket says 172.21.15.77:60964) 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e2ffd640 1 -- 172.21.15.77:0/2185124144 learned_addr learned my addr 172.21.15.77:0/2185124144 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91dbfff640 1 --1- 172.21.15.77:0/2185124144 >> v1:172.21.15.77:6789/0 conn(0x7f91e40b0d80 0x7f91e40cff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47460/0 (socket says 172.21.15.77:47460) 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e37fe640 1 --2- 172.21.15.77:0/2185124144 >> v2:172.21.15.121:3300/0 conn(0x7f91e40d27a0 0x7f91e40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e2ffd640 1 -- 172.21.15.77:0/2185124144 >> v2:172.21.15.121:3300/0 conn(0x7f91e40d27a0 msgr2=0x7f91e40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e2ffd640 1 --2- 172.21.15.77:0/2185124144 >> v2:172.21.15.121:3300/0 conn(0x7f91e40d27a0 0x7f91e40cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e2ffd640 1 -- 172.21.15.77:0/2185124144 >> v1:172.21.15.77:6789/0 conn(0x7f91e40b0d80 legacy=0x7f91e40cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e27fc640 1 -- 172.21.15.77:0/2185124144 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2211743086 0 0) 0x7f91e40cf9f0 con 0x7f91e40b0d80 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.386+0000 7f91e2ffd640 1 -- 172.21.15.77:0/2185124144 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e40cf360 con 0x7f91e415c090 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e2ffd640 1 --2- 172.21.15.77:0/2185124144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91e415c090 0x7f91e40b1180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f91e40cde20 tx=0x7f91d402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723b57e2e3224e09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e27fc640 1 -- 172.21.15.77:0/2185124144 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4037070 con 0x7f91e415c090 2024-04-25T01:21:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e27fc640 1 -- 172.21.15.77:0/2185124144 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91d402ae50 con 0x7f91e415c090 2024-04-25T01:21:57.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e27fc640 1 -- 172.21.15.77:0/2185124144 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4032860 con 0x7f91e415c090 2024-04-25T01:21:57.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e9810640 1 -- 172.21.15.77:0/2185124144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91e415c090 msgr2=0x7f91e40b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e9810640 1 --2- 172.21.15.77:0/2185124144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91e415c090 0x7f91e40b1180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f91e40cde20 tx=0x7f91d402a300 comp rx=0 tx=0).stop 2024-04-25T01:21:57.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e9810640 1 -- 172.21.15.77:0/2185124144 shutdown_connections 2024-04-25T01:21:57.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e9810640 1 --2- 172.21.15.77:0/2185124144 >> v2:172.21.15.121:3300/0 conn(0x7f91e40d27a0 0x7f91e40cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e9810640 1 --2- 172.21.15.77:0/2185124144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91e415c090 0x7f91e40b1180 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e9810640 1 -- 172.21.15.77:0/2185124144 >> 172.21.15.77:0/2185124144 conn(0x7f91e40d1c80 msgr2=0x7f91e40d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e9810640 1 -- 172.21.15.77:0/2185124144 shutdown_connections 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.387+0000 7f91e9810640 1 -- 172.21.15.77:0/2185124144 wait complete. 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.388+0000 7f91e9810640 1 Processor -- start 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91e9810640 1 -- start start 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91e9810640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91e40b0d80 0x7f91e414b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91e9810640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e415c090 0x7f91e40b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91e9810640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91e40b6020 0x7f91e40c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91e9810640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f91e40d0620 con 0x7f91e40b0d80 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91e9810640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f91e415c470 con 0x7f91e40b6020 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91e9810640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f91e40d07a0 con 0x7f91e415c090 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91e2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91e40b0d80 0x7f91e414b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91e2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91e40b0d80 0x7f91e414b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60966/0 (socket says 172.21.15.77:60966) 2024-04-25T01:21:57.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91e2ffd640 1 -- 172.21.15.77:0/707884161 learned_addr learned my addr 172.21.15.77:0/707884161 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91e37fe640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91e40b6020 0x7f91e40c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.389+0000 7f91dbfff640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e415c090 0x7f91e40b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e2ffd640 1 -- 172.21.15.77:0/707884161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e415c090 msgr2=0x7f91e40b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e2ffd640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e415c090 0x7f91e40b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e2ffd640 1 -- 172.21.15.77:0/707884161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91e40b6020 msgr2=0x7f91e40c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e2ffd640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91e40b6020 0x7f91e40c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e2ffd640 1 -- 172.21.15.77:0/707884161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e40b3a80 con 0x7f91e40b0d80 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e37fe640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91e40b6020 0x7f91e40c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e2ffd640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91e40b0d80 0x7f91e414b090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f91d402a340 tx=0x7f91d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e0ff9640 1 -- 172.21.15.77:0/707884161 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4041070 con 0x7f91e40b0d80 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e0ff9640 1 -- 172.21.15.77:0/707884161 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91d403c670 con 0x7f91e40b0d80 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e9810640 1 -- 172.21.15.77:0/707884161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e40b66c0 con 0x7f91e40b0d80 2024-04-25T01:21:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e0ff9640 1 -- 172.21.15.77:0/707884161 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4037040 con 0x7f91e40b0d80 2024-04-25T01:21:57.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.390+0000 7f91e9810640 1 -- 172.21.15.77:0/707884161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e40b6c80 con 0x7f91e40b0d80 2024-04-25T01:21:57.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.391+0000 7f91e9810640 1 -- 172.21.15.77:0/707884161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e40b6430 con 0x7f91e40b0d80 2024-04-25T01:21:57.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.391+0000 7f91e0ff9640 1 -- 172.21.15.77:0/707884161 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f91d4044070 con 0x7f91e40b0d80 2024-04-25T01:21:57.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.392+0000 7f91e0ff9640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f91c406be30 0x7f91c406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.392+0000 7f91e0ff9640 1 -- 172.21.15.77:0/707884161 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f91d40abb90 con 0x7f91e40b0d80 2024-04-25T01:21:57.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.392+0000 7f91dbfff640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f91c406be30 0x7f91c406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.392+0000 7f91dbfff640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f91c406be30 0x7f91c406e2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f91e41f5350 tx=0x7f91cc007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.396+0000 7f91e0ff9640 1 -- 172.21.15.77:0/707884161 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f91d407a290 con 0x7f91e40b0d80 2024-04-25T01:21:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.570+0000 7f91e9810640 1 -- 172.21.15.77:0/707884161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91e40d10c0 con 0x7f91e40b0d80 2024-04-25T01:21:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.571+0000 7f91e0ff9640 1 -- 172.21.15.77:0/707884161 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f91d4079b90 con 0x7f91e40b0d80 2024-04-25T01:21:57.553 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:21:57.554 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.573+0000 7f91e9810640 1 -- 172.21.15.77:0/707884161 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f91c406be30 msgr2=0x7f91c406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.573+0000 7f91e9810640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f91c406be30 0x7f91c406e2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f91e41f5350 tx=0x7f91cc007b50 comp rx=0 tx=0).stop 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.574+0000 7f91e9810640 1 -- 172.21.15.77:0/707884161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91e40b0d80 msgr2=0x7f91e414b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.574+0000 7f91e9810640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91e40b0d80 0x7f91e414b090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f91d402a340 tx=0x7f91d40046c0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.574+0000 7f91e9810640 1 -- 172.21.15.77:0/707884161 shutdown_connections 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.574+0000 7f91e9810640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f91c406be30 0x7f91c406e2f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.574+0000 7f91e9810640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91e40b6020 0x7f91e40c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.574+0000 7f91e9810640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e415c090 0x7f91e40b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.574+0000 7f91e9810640 1 --2- 172.21.15.77:0/707884161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91e40b0d80 0x7f91e414b090 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.574+0000 7f91e9810640 1 -- 172.21.15.77:0/707884161 >> 172.21.15.77:0/707884161 conn(0x7f91e40d1c80 msgr2=0x7f91e40af900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.574+0000 7f91e9810640 1 -- 172.21.15.77:0/707884161 shutdown_connections 2024-04-25T01:21:57.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.574+0000 7f91e9810640 1 -- 172.21.15.77:0/707884161 wait complete. 2024-04-25T01:21:57.567 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:21:57.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:21:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.684+0000 7efddeaa5640 1 Processor -- start 2024-04-25T01:21:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.684+0000 7efddeaa5640 1 -- start start 2024-04-25T01:21:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.685+0000 7efddeaa5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdd806b830 0x7efdd815d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.685+0000 7efddeaa5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdd80b0200 con 0x7efdd80af8c0 2024-04-25T01:21:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.685+0000 7efddeaa5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdd80b5840 con 0x7efdd80cd880 2024-04-25T01:21:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.685+0000 7efddeaa5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efdd80b5b20 con 0x7efdd806b830 2024-04-25T01:21:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.685+0000 7efddc841640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efdd80af8c0 0x7efdd80b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47470/0 (socket says 172.21.15.77:47470) 2024-04-25T01:21:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.685+0000 7efddc841640 1 -- 172.21.15.77:0/1968988919 learned_addr learned my addr 172.21.15.77:0/1968988919 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:57.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.685+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1956738112 0 0) 0x7efdd80b0200 con 0x7efdd80af8c0 2024-04-25T01:21:57.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.685+0000 7efddd042640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdd806b830 0x7efdd815d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.685+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdc0003520 con 0x7efdd80af8c0 2024-04-25T01:21:57.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1452243483 0 0) 0x7efdc0003520 con 0x7efdd80af8c0 2024-04-25T01:21:57.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdd806b830 msgr2=0x7efdd815d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:57.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdd806b830 0x7efdd815d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 >> v1:172.21.15.121:6789/0 conn(0x7efdd80cd880 legacy=0x7efdd806b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd80b5e00 con 0x7efdd80af8c0 2024-04-25T01:21:57.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7efdc8001670 con 0x7efdd80af8c0 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 >> v1:172.21.15.77:6789/0 conn(0x7efdd80af8c0 legacy=0x7efdd80b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdd806b830 0x7efdc00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc0004010 0x7efdc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc0004d40 0x7efdc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efdd80b5b20 con 0x7efdc0004010 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efdc000bb80 con 0x7efdc0004d40 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efdc000bd80 con 0x7efdd806b830 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd7fff640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc0004010 0x7efdc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efddc841640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdd806b830 0x7efdc00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efddd042640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc0004d40 0x7efdc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd7fff640 1 -- 172.21.15.77:0/1968988919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdd806b830 msgr2=0x7efdc00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd7fff640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdd806b830 0x7efdc00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd7fff640 1 -- 172.21.15.77:0/1968988919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc0004d40 msgr2=0x7efdc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd7fff640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc0004d40 0x7efdc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.686+0000 7efdd7fff640 1 -- 172.21.15.77:0/1968988919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc0003520 con 0x7efdc0004010 2024-04-25T01:21:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.687+0000 7efdd7fff640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc0004010 0x7efdc0004800 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efdcc001750 tx=0x7efdc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e48ba393875a98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.687+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc001370 con 0x7efdc0004010 2024-04-25T01:21:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.687+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdcc0052d0 con 0x7efdc0004010 2024-04-25T01:21:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.687+0000 7efdd77fe640 1 -- 172.21.15.77:0/1968988919 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc02abe0 con 0x7efdc0004010 2024-04-25T01:21:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.687+0000 7efddeaa5640 1 -- 172.21.15.77:0/1968988919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc0004010 msgr2=0x7efdc0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.687+0000 7efddeaa5640 1 --2- 172.21.15.77:0/1968988919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc0004010 0x7efdc0004800 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efdcc001750 tx=0x7efdc0005830 comp rx=0 tx=0).stop 2024-04-25T01:21:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.688+0000 7efddd042640 1 -- 172.21.15.77:0/1968988919 reap_dead start 2024-04-25T01:21:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.688+0000 7efddeaa5640 1 -- 172.21.15.77:0/1968988919 shutdown_connections 2024-04-25T01:21:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.688+0000 7efddeaa5640 1 -- 172.21.15.77:0/1968988919 >> 172.21.15.77:0/1968988919 conn(0x7efdd80b1f00 msgr2=0x7efdd80b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.688+0000 7efddeaa5640 1 -- 172.21.15.77:0/1968988919 shutdown_connections 2024-04-25T01:21:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.688+0000 7efddeaa5640 1 -- 172.21.15.77:0/1968988919 wait complete. 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.689+0000 7efddeaa5640 1 Processor -- start 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.689+0000 7efddeaa5640 1 -- start start 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddeaa5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0004010 0x7efdd81f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddeaa5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdd815eb60 0x7efdd80cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddeaa5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdd815cd10 0x7efdd80c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddeaa5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efdd80afe20 con 0x7efdd815cd10 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddeaa5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efdd80afca0 con 0x7efdd815eb60 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddeaa5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efdd80ca980 con 0x7efdc0004010 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddd042640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdd815cd10 0x7efdd80c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddd042640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdd815cd10 0x7efdd80c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60970/0 (socket says 172.21.15.77:60970) 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddd042640 1 -- 172.21.15.77:0/2732950437 learned_addr learned my addr 172.21.15.77:0/2732950437 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddc841640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0004010 0x7efdd81f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efdd7fff640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdd815eb60 0x7efdd80cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddd042640 1 -- 172.21.15.77:0/2732950437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0004010 msgr2=0x7efdd81f69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddd042640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0004010 0x7efdd81f69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddd042640 1 -- 172.21.15.77:0/2732950437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdd815eb60 msgr2=0x7efdd80cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddd042640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdd815eb60 0x7efdd80cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddd042640 1 -- 172.21.15.77:0/2732950437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd815e030 con 0x7efdd815cd10 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddc841640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0004010 0x7efdd81f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddd042640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdd815cd10 0x7efdd80c95e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efdd80b5770 tx=0x7efdd0007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efdd57fa640 1 -- 172.21.15.77:0/2732950437 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd00046a0 con 0x7efdd815cd10 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efdd57fa640 1 -- 172.21.15.77:0/2732950437 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd001a070 con 0x7efdd815cd10 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddeaa5640 1 -- 172.21.15.77:0/2732950437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd80c79e0 con 0x7efdd815cd10 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efdd57fa640 1 -- 172.21.15.77:0/2732950437 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd0015640 con 0x7efdd815cd10 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.690+0000 7efddeaa5640 1 -- 172.21.15.77:0/2732950437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd80c7fa0 con 0x7efdd815cd10 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.691+0000 7efddeaa5640 1 -- 172.21.15.77:0/2732950437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd8068fb0 con 0x7efdd815cd10 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.692+0000 7efdd57fa640 1 -- 172.21.15.77:0/2732950437 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efdd00157e0 con 0x7efdd815cd10 2024-04-25T01:21:57.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.692+0000 7efdd57fa640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efdc406be30 0x7efdc406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.692+0000 7efdd57fa640 1 -- 172.21.15.77:0/2732950437 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7efdd008b850 con 0x7efdd815cd10 2024-04-25T01:21:57.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.692+0000 7efddc841640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efdc406be30 0x7efdc406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.693+0000 7efddc841640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efdc406be30 0x7efdc406e2f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efdd815b9f0 tx=0x7efdc8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.696+0000 7efdd57fa640 1 -- 172.21.15.77:0/2732950437 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7efdd0024020 con 0x7efdd815cd10 2024-04-25T01:21:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.866+0000 7efddeaa5640 1 -- 172.21.15.77:0/2732950437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efdd8074750 con 0x7efdd815cd10 2024-04-25T01:21:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.866+0000 7efdd57fa640 1 -- 172.21.15.77:0/2732950437 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7efdd0059f50 con 0x7efdd815cd10 2024-04-25T01:21:57.848 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:21:57.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 -- 172.21.15.77:0/2732950437 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efdc406be30 msgr2=0x7efdc406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efdc406be30 0x7efdc406e2f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efdd815b9f0 tx=0x7efdc8009040 comp rx=0 tx=0).stop 2024-04-25T01:21:57.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 -- 172.21.15.77:0/2732950437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdd815cd10 msgr2=0x7efdd80c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdd815cd10 0x7efdd80c95e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efdd80b5770 tx=0x7efdd0007700 comp rx=0 tx=0).stop 2024-04-25T01:21:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 -- 172.21.15.77:0/2732950437 shutdown_connections 2024-04-25T01:21:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efdc406be30 0x7efdc406e2f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdd815cd10 0x7efdd80c95e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdd815eb60 0x7efdd80cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 --2- 172.21.15.77:0/2732950437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0004010 0x7efdd81f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 -- 172.21.15.77:0/2732950437 >> 172.21.15.77:0/2732950437 conn(0x7efdd80b1f00 msgr2=0x7efdd8072890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 -- 172.21.15.77:0/2732950437 shutdown_connections 2024-04-25T01:21:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.868+0000 7efddeaa5640 1 -- 172.21.15.77:0/2732950437 wait complete. 2024-04-25T01:21:57.861 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:21:57.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.978+0000 7f1f83efc640 1 Processor -- start 2024-04-25T01:21:57.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.979+0000 7f1f83efc640 1 -- start start 2024-04-25T01:21:57.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.979+0000 7f1f83efc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1f7c1619d0 0x7f1f7c0d49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.979+0000 7f1f83efc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1f7c0d1c80 0x7f1f7c0d2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.979+0000 7f1f83efc640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f7c0cfe30 con 0x7f1f7c0d29c0 2024-04-25T01:21:57.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.979+0000 7f1f83efc640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f1f7c0ceab0 con 0x7f1f7c1619d0 2024-04-25T01:21:57.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.979+0000 7f1f83efc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1f7c0ced90 con 0x7f1f7c0d1c80 2024-04-25T01:21:57.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.980+0000 7f1f81c98640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1f7c1619d0 0x7f1f7c0d49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.980+0000 7f1f81c98640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1f7c1619d0 0x7f1f7c0d49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60986/0 (socket says 172.21.15.77:60986) 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.980+0000 7f1f81c98640 1 -- 172.21.15.77:0/193266773 learned_addr learned my addr 172.21.15.77:0/193266773 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.980+0000 7f1f81497640 1 --2- 172.21.15.77:0/193266773 >> v2:172.21.15.121:3300/0 conn(0x7f1f7c0d1c80 0x7f1f7c0d2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.980+0000 7f1f81c98640 1 -- 172.21.15.77:0/193266773 >> v2:172.21.15.121:3300/0 conn(0x7f1f7c0d1c80 msgr2=0x7f1f7c0d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.980+0000 7f1f81c98640 1 --2- 172.21.15.77:0/193266773 >> v2:172.21.15.121:3300/0 conn(0x7f1f7c0d1c80 0x7f1f7c0d2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.980+0000 7f1f81c98640 1 -- 172.21.15.77:0/193266773 >> v1:172.21.15.121:6789/0 conn(0x7f1f7c0d29c0 legacy=0x7f1f7c161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.980+0000 7f1f81c98640 1 -- 172.21.15.77:0/193266773 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c0cf070 con 0x7f1f7c1619d0 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.980+0000 7f1f81c98640 1 --2- 172.21.15.77:0/193266773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f7c1619d0 0x7f1f7c0d49c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0b6a90 tx=0x7f1f7802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ede2785ba717076 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.981+0000 7f1f80c96640 1 -- 172.21.15.77:0/193266773 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f78036070 con 0x7f1f7c1619d0 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.981+0000 7f1f80c96640 1 -- 172.21.15.77:0/193266773 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f78031600 con 0x7f1f7c1619d0 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.981+0000 7f1f80c96640 1 -- 172.21.15.77:0/193266773 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f780319d0 con 0x7f1f7c1619d0 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.981+0000 7f1f83efc640 1 -- 172.21.15.77:0/193266773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f7c1619d0 msgr2=0x7f1f7c0d49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.981+0000 7f1f83efc640 1 --2- 172.21.15.77:0/193266773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f7c1619d0 0x7f1f7c0d49c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0b6a90 tx=0x7f1f7802a6f0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.981+0000 7f1f83efc640 1 -- 172.21.15.77:0/193266773 shutdown_connections 2024-04-25T01:21:57.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.981+0000 7f1f83efc640 1 --2- 172.21.15.77:0/193266773 >> v2:172.21.15.121:3300/0 conn(0x7f1f7c0d1c80 0x7f1f7c0d2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.981+0000 7f1f83efc640 1 --2- 172.21.15.77:0/193266773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f7c1619d0 0x7f1f7c0d49c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.981+0000 7f1f83efc640 1 -- 172.21.15.77:0/193266773 >> 172.21.15.77:0/193266773 conn(0x7f1f7c0d3ea0 msgr2=0x7f1f7c0d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:57.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.981+0000 7f1f83efc640 1 -- 172.21.15.77:0/193266773 shutdown_connections 2024-04-25T01:21:57.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.981+0000 7f1f83efc640 1 -- 172.21.15.77:0/193266773 wait complete. 2024-04-25T01:21:57.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.982+0000 7f1f83efc640 1 Processor -- start 2024-04-25T01:21:57.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.982+0000 7f1f83efc640 1 -- start start 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f83efc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f7c0d1c80 0x7f1f7c0cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f83efc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f7c0d29c0 0x7f1f7c15b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f83efc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f7c1619d0 0x7f1f7c15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f83efc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c0cf910 con 0x7f1f7c0d1c80 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f83efc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c0cf630 con 0x7f1f7c1619d0 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f83efc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f7c0b4390 con 0x7f1f7c0d29c0 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f81c98640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f7c0d1c80 0x7f1f7c0cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f81c98640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f7c0d1c80 0x7f1f7c0cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60990/0 (socket says 172.21.15.77:60990) 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f81c98640 1 -- 172.21.15.77:0/1655814494 learned_addr learned my addr 172.21.15.77:0/1655814494 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f81497640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f7c0d29c0 0x7f1f7c15b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f82499640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f7c1619d0 0x7f1f7c15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f81c98640 1 -- 172.21.15.77:0/1655814494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f7c0d29c0 msgr2=0x7f1f7c15b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f81c98640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f7c0d29c0 0x7f1f7c15b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f81c98640 1 -- 172.21.15.77:0/1655814494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f7c1619d0 msgr2=0x7f1f7c15b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f81c98640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f7c1619d0 0x7f1f7c15b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.983+0000 7f1f81c98640 1 -- 172.21.15.77:0/1655814494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c1603f0 con 0x7f1f7c0d1c80 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.984+0000 7f1f81c98640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f7c0d1c80 0x7f1f7c0cc9d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f780093e0 tx=0x7f1f78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.984+0000 7f1f727fc640 1 -- 172.21.15.77:0/1655814494 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f78041020 con 0x7f1f7c0d1c80 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.984+0000 7f1f727fc640 1 -- 172.21.15.77:0/1655814494 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f78007820 con 0x7f1f7c0d1c80 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.984+0000 7f1f83efc640 1 -- 172.21.15.77:0/1655814494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7c15f7e0 con 0x7f1f7c0d1c80 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.984+0000 7f1f727fc640 1 -- 172.21.15.77:0/1655814494 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f78036040 con 0x7f1f7c0d1c80 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.984+0000 7f1f83efc640 1 -- 172.21.15.77:0/1655814494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f7c15fda0 con 0x7f1f7c0d1c80 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.985+0000 7f1f83efc640 1 -- 172.21.15.77:0/1655814494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f7c068180 con 0x7f1f7c0d1c80 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.985+0000 7f1f727fc640 1 -- 172.21.15.77:0/1655814494 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1f78044020 con 0x7f1f7c0d1c80 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.986+0000 7f1f727fc640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6006be30 0x7f1f6006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.986+0000 7f1f727fc640 1 -- 172.21.15.77:0/1655814494 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1f780abfd0 con 0x7f1f7c0d1c80 2024-04-25T01:21:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.986+0000 7f1f81497640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6006be30 0x7f1f6006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:57.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.986+0000 7f1f81497640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6006be30 0x7f1f6006e2f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c006fd0 tx=0x7f1f6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:57.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:57.990+0000 7f1f727fc640 1 -- 172.21.15.77:0/1655814494 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1f7804b030 con 0x7f1f7c0d1c80 2024-04-25T01:21:58.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.171+0000 7f1f83efc640 1 -- 172.21.15.77:0/1655814494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1f7c0b5770 con 0x7f1f7c0d1c80 2024-04-25T01:21:58.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.171+0000 7f1f727fc640 1 -- 172.21.15.77:0/1655814494 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f1f7c0cfe30 con 0x7f1f7c0d1c80 2024-04-25T01:21:58.153 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:21:58.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 -- 172.21.15.77:0/1655814494 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6006be30 msgr2=0x7f1f6006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6006be30 0x7f1f6006e2f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c006fd0 tx=0x7f1f6c008040 comp rx=0 tx=0).stop 2024-04-25T01:21:58.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 -- 172.21.15.77:0/1655814494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f7c0d1c80 msgr2=0x7f1f7c0cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f7c0d1c80 0x7f1f7c0cc9d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f780093e0 tx=0x7f1f78004a40 comp rx=0 tx=0).stop 2024-04-25T01:21:58.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 -- 172.21.15.77:0/1655814494 shutdown_connections 2024-04-25T01:21:58.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6006be30 0x7f1f6006e2f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f7c1619d0 0x7f1f7c15b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f7c0d29c0 0x7f1f7c15b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 --2- 172.21.15.77:0/1655814494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f7c0d1c80 0x7f1f7c0cc9d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 -- 172.21.15.77:0/1655814494 >> 172.21.15.77:0/1655814494 conn(0x7f1f7c0d3ea0 msgr2=0x7f1f7c0ca240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:58.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 -- 172.21.15.77:0/1655814494 shutdown_connections 2024-04-25T01:21:58.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.173+0000 7f1f83efc640 1 -- 172.21.15.77:0/1655814494 wait complete. 2024-04-25T01:21:58.167 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:21:58.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.284+0000 7fb2fe309640 1 Processor -- start 2024-04-25T01:21:58.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.284+0000 7fb2fe309640 1 -- start start 2024-04-25T01:21:58.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.285+0000 7fb2fe309640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f815e460 0x7fb2f80d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.285+0000 7fb2fe309640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2f80d0de0 con 0x7fb2f80cebe0 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.285+0000 7fb2fe309640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2f8070280 con 0x7fb2f80d09a0 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.285+0000 7fb2fe309640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f806c230 con 0x7fb2f815e460 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.285+0000 7fb2fc8a6640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb2f80cebe0 0x7fb2f80ccf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47482/0 (socket says 172.21.15.77:47482) 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.285+0000 7fb2fc8a6640 1 -- 172.21.15.77:0/153885933 learned_addr learned my addr 172.21.15.77:0/153885933 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.285+0000 7fb2f7fff640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f815e460 0x7fb2f80d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2374624832 0 0) 0x7fb2f80d0de0 con 0x7fb2f80cebe0 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2e0003520 con 0x7fb2f80cebe0 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388577693 0 0) 0x7fb2f8070280 con 0x7fb2f80d09a0 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2f80d0de0 con 0x7fb2f80d09a0 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 721006764 0 0) 0x7fb2e0003520 con 0x7fb2f80cebe0 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f815e460 msgr2=0x7fb2f80d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f815e460 0x7fb2f80d2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 >> v1:172.21.15.121:6789/0 conn(0x7fb2f80d09a0 legacy=0x7fb2f80ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f80d1080 con 0x7fb2f80cebe0 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fb2f00019f0 con 0x7fb2f80cebe0 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f7fff640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f815e460 0x7fb2f80d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 >> v1:172.21.15.77:6789/0 conn(0x7fb2f80cebe0 legacy=0x7fb2f80ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2e0003b70 0x7fb2e0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f815e460 0x7fb2e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2e0004d40 0x7fb2e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e000ba70 con 0x7fb2e0004d40 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e000bbf0 con 0x7fb2e0003b70 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb2e000bd70 con 0x7fb2f815e460 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2fc8a6640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2e0004d40 0x7fb2e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f7fff640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2e0003b70 0x7fb2e0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2f77fe640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f815e460 0x7fb2e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2fc8a6640 1 -- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f815e460 msgr2=0x7fb2e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2fc8a6640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f815e460 0x7fb2e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2fc8a6640 1 -- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2e0003b70 msgr2=0x7fb2e0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2fc8a6640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2e0003b70 0x7fb2e0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.286+0000 7fb2fc8a6640 1 -- 172.21.15.77:0/153885933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e0003520 con 0x7fb2e0004d40 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.287+0000 7fb2fc8a6640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2e0004d40 0x7fb2e00051c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000a800 tx=0x7fb2e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425750dcdb9f7fe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.287+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f000a430 con 0x7fb2e0004d40 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.287+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f002f070 con 0x7fb2e0004d40 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.287+0000 7fb2f6ffd640 1 -- 172.21.15.77:0/153885933 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f002bbf0 con 0x7fb2e0004d40 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.288+0000 7fb2fe309640 1 -- 172.21.15.77:0/153885933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2e0004d40 msgr2=0x7fb2e00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.288+0000 7fb2fe309640 1 --2- 172.21.15.77:0/153885933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2e0004d40 0x7fb2e00051c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000a800 tx=0x7fb2e0005830 comp rx=0 tx=0).stop 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.288+0000 7fb2fc8a6640 1 -- 172.21.15.77:0/153885933 reap_dead start 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.288+0000 7fb2fe309640 1 -- 172.21.15.77:0/153885933 shutdown_connections 2024-04-25T01:21:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.288+0000 7fb2fe309640 1 -- 172.21.15.77:0/153885933 >> 172.21.15.77:0/153885933 conn(0x7fb2f80d1c50 msgr2=0x7fb2f80d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:58.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.288+0000 7fb2fe309640 1 -- 172.21.15.77:0/153885933 shutdown_connections 2024-04-25T01:21:58.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.288+0000 7fb2fe309640 1 -- 172.21.15.77:0/153885933 wait complete. 2024-04-25T01:21:58.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.289+0000 7fb2fe309640 1 Processor -- start 2024-04-25T01:21:58.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.289+0000 7fb2fe309640 1 -- start start 2024-04-25T01:21:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2fe309640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f80ca230 0x7fb2f814df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2fe309640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f809ba90 0x7fb2f809bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2fe309640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f809ec90 0x7fb2f809f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2fe309640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f80cc720 con 0x7fb2f809ba90 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2fe309640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f80cca90 con 0x7fb2f80ca230 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2fe309640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f80d2770 con 0x7fb2f809ec90 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f809ba90 0x7fb2f809bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f809ba90 0x7fb2f809bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32786/0 (socket says 172.21.15.77:32786) 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f77fe640 1 -- 172.21.15.77:0/3022097452 learned_addr learned my addr 172.21.15.77:0/3022097452 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f77fe640 1 -- 172.21.15.77:0/3022097452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f809ec90 msgr2=0x7fb2f809f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2fc8a6640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f809ec90 0x7fb2f809f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f7fff640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f80ca230 0x7fb2f814df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f77fe640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f809ec90 0x7fb2f809f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f77fe640 1 -- 172.21.15.77:0/3022097452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f80ca230 msgr2=0x7fb2f814df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f77fe640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f80ca230 0x7fb2f814df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f77fe640 1 -- 172.21.15.77:0/3022097452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f809dc60 con 0x7fb2f809ba90 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f7fff640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f80ca230 0x7fb2f814df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2fc8a6640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f809ec90 0x7fb2f809f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f77fe640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f809ba90 0x7fb2f809bf10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00f7d0 tx=0x7fb2ec00fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f4ff9640 1 -- 172.21.15.77:0/3022097452 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec00fe80 con 0x7fb2f809ba90 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f4ff9640 1 -- 172.21.15.77:0/3022097452 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2ec018070 con 0x7fb2f809ba90 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2f4ff9640 1 -- 172.21.15.77:0/3022097452 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec00d730 con 0x7fb2f809ba90 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2fe309640 1 -- 172.21.15.77:0/3022097452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f814e4a0 con 0x7fb2f809ba90 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.290+0000 7fb2fe309640 1 -- 172.21.15.77:0/3022097452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f809e1b0 con 0x7fb2f809ba90 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.291+0000 7fb2fe309640 1 -- 172.21.15.77:0/3022097452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f8068fb0 con 0x7fb2f809ba90 2024-04-25T01:21:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.292+0000 7fb2f4ff9640 1 -- 172.21.15.77:0/3022097452 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb2ec00d8d0 con 0x7fb2f809ba90 2024-04-25T01:21:58.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.292+0000 7fb2f4ff9640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb2e406be30 0x7fb2e406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.292+0000 7fb2f4ff9640 1 -- 172.21.15.77:0/3022097452 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb2ec08b130 con 0x7fb2f809ba90 2024-04-25T01:21:58.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.292+0000 7fb2f7fff640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb2e406be30 0x7fb2e406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.293+0000 7fb2f7fff640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb2e406be30 0x7fb2e406e2f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009ab0 tx=0x7fb2e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:58.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.296+0000 7fb2f4ff9640 1 -- 172.21.15.77:0/3022097452 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb2ec01a310 con 0x7fb2f809ba90 2024-04-25T01:21:58.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.468+0000 7fb2fe309640 1 -- 172.21.15.77:0/3022097452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb2f80ce670 con 0x7fb2f809ba90 2024-04-25T01:21:58.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.468+0000 7fb2f4ff9640 1 -- 172.21.15.77:0/3022097452 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7fb2ec0597a0 con 0x7fb2f809ba90 2024-04-25T01:21:58.450 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:21:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 -- 172.21.15.77:0/3022097452 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb2e406be30 msgr2=0x7fb2e406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb2e406be30 0x7fb2e406e2f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009ab0 tx=0x7fb2e8008040 comp rx=0 tx=0).stop 2024-04-25T01:21:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 -- 172.21.15.77:0/3022097452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f809ba90 msgr2=0x7fb2f809bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f809ba90 0x7fb2f809bf10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00f7d0 tx=0x7fb2ec00fc90 comp rx=0 tx=0).stop 2024-04-25T01:21:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 -- 172.21.15.77:0/3022097452 shutdown_connections 2024-04-25T01:21:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb2e406be30 0x7fb2e406e2f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f809ba90 0x7fb2f809bf10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f809ec90 0x7fb2f809f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 --2- 172.21.15.77:0/3022097452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f80ca230 0x7fb2f814df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 -- 172.21.15.77:0/3022097452 >> 172.21.15.77:0/3022097452 conn(0x7fb2f80d1c50 msgr2=0x7fb2f80d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 -- 172.21.15.77:0/3022097452 shutdown_connections 2024-04-25T01:21:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.470+0000 7fb2fe309640 1 -- 172.21.15.77:0/3022097452 wait complete. 2024-04-25T01:21:58.463 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:21:58.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.580+0000 7fcb2769a640 1 Processor -- start 2024-04-25T01:21:58.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.580+0000 7fcb2769a640 1 -- start start 2024-04-25T01:21:58.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.581+0000 7fcb2769a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb200700c0 0x7fcb2015d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.581+0000 7fcb2769a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb200cedb0 0x7fcb2015dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.581+0000 7fcb2769a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb200d00d0 con 0x7fcb200cdc50 2024-04-25T01:21:58.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.581+0000 7fcb2769a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fcb200cd350 con 0x7fcb200700c0 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.581+0000 7fcb2769a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcb200d03b0 con 0x7fcb200cedb0 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.581+0000 7fcb24c35640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb200700c0 0x7fcb2015d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.581+0000 7fcb24c35640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb200700c0 0x7fcb2015d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32802/0 (socket says 172.21.15.77:32802) 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.581+0000 7fcb24c35640 1 -- 172.21.15.77:0/2223814057 learned_addr learned my addr 172.21.15.77:0/2223814057 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.581+0000 7fcb25c37640 1 --2- 172.21.15.77:0/2223814057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb200cedb0 0x7fcb2015dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.582+0000 7fcb24c35640 1 -- 172.21.15.77:0/2223814057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb200cedb0 msgr2=0x7fcb2015dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.582+0000 7fcb24c35640 1 --2- 172.21.15.77:0/2223814057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb200cedb0 0x7fcb2015dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.582+0000 7fcb24c35640 1 -- 172.21.15.77:0/2223814057 >> v1:172.21.15.121:6789/0 conn(0x7fcb200cdc50 legacy=0x7fcb200ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.582+0000 7fcb24c35640 1 -- 172.21.15.77:0/2223814057 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb200d0690 con 0x7fcb200700c0 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.582+0000 7fcb24c35640 1 --2- 172.21.15.77:0/2223814057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb200700c0 0x7fcb2015d960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009870 tx=0x7fcb1002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ab79100a878dfa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.582+0000 7fcb17fff640 1 -- 172.21.15.77:0/2223814057 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb10037070 con 0x7fcb200700c0 2024-04-25T01:21:58.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.582+0000 7fcb17fff640 1 -- 172.21.15.77:0/2223814057 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb10032440 con 0x7fcb200700c0 2024-04-25T01:21:58.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.583+0000 7fcb17fff640 1 -- 172.21.15.77:0/2223814057 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb10032810 con 0x7fcb200700c0 2024-04-25T01:21:58.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.583+0000 7fcb2769a640 1 -- 172.21.15.77:0/2223814057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb200700c0 msgr2=0x7fcb2015d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.583+0000 7fcb2769a640 1 --2- 172.21.15.77:0/2223814057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb200700c0 0x7fcb2015d960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009870 tx=0x7fcb1002a1c0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.583+0000 7fcb2769a640 1 -- 172.21.15.77:0/2223814057 shutdown_connections 2024-04-25T01:21:58.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.583+0000 7fcb2769a640 1 --2- 172.21.15.77:0/2223814057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb200cedb0 0x7fcb2015dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.583+0000 7fcb2769a640 1 --2- 172.21.15.77:0/2223814057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb200700c0 0x7fcb2015d960 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.583+0000 7fcb2769a640 1 -- 172.21.15.77:0/2223814057 >> 172.21.15.77:0/2223814057 conn(0x7fcb200d1cf0 msgr2=0x7fcb200d2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:58.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.583+0000 7fcb2769a640 1 -- 172.21.15.77:0/2223814057 shutdown_connections 2024-04-25T01:21:58.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.583+0000 7fcb2769a640 1 -- 172.21.15.77:0/2223814057 wait complete. 2024-04-25T01:21:58.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.584+0000 7fcb2769a640 1 Processor -- start 2024-04-25T01:21:58.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.584+0000 7fcb2769a640 1 -- start start 2024-04-25T01:21:58.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb2769a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb200700c0 0x7fcb2014a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb2769a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb200cdc50 0x7fcb200c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb2769a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb200cedb0 0x7fcb200c6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb2769a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcb200d1170 con 0x7fcb200700c0 2024-04-25T01:21:58.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb2769a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcb200d0c50 con 0x7fcb200cedb0 2024-04-25T01:21:58.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb2769a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcb2015b7f0 con 0x7fcb200cdc50 2024-04-25T01:21:58.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb25436640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb200700c0 0x7fcb2014a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb25436640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb200700c0 0x7fcb2014a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32804/0 (socket says 172.21.15.77:32804) 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb25436640 1 -- 172.21.15.77:0/3141456067 learned_addr learned my addr 172.21.15.77:0/3141456067 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb25c37640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb200cedb0 0x7fcb200c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb24c35640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb200cdc50 0x7fcb200c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb25436640 1 -- 172.21.15.77:0/3141456067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb200cdc50 msgr2=0x7fcb200c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb25436640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb200cdc50 0x7fcb200c6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb25436640 1 -- 172.21.15.77:0/3141456067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb200cedb0 msgr2=0x7fcb200c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb25436640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb200cedb0 0x7fcb200c6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb25436640 1 -- 172.21.15.77:0/3141456067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb200c6d80 con 0x7fcb200700c0 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.585+0000 7fcb25436640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb200700c0 0x7fcb2014a0a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00c790 tx=0x7fcb1c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.586+0000 7fcb15ffb640 1 -- 172.21.15.77:0/3141456067 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c016020 con 0x7fcb200700c0 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.586+0000 7fcb15ffb640 1 -- 172.21.15.77:0/3141456067 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb1c006d50 con 0x7fcb200700c0 2024-04-25T01:21:58.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.586+0000 7fcb2769a640 1 -- 172.21.15.77:0/3141456067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb200c7060 con 0x7fcb200700c0 2024-04-25T01:21:58.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.586+0000 7fcb15ffb640 1 -- 172.21.15.77:0/3141456067 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c003bb0 con 0x7fcb200700c0 2024-04-25T01:21:58.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.586+0000 7fcb2769a640 1 -- 172.21.15.77:0/3141456067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb2015b480 con 0x7fcb200700c0 2024-04-25T01:21:58.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.587+0000 7fcb2769a640 1 -- 172.21.15.77:0/3141456067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb20068fb0 con 0x7fcb200700c0 2024-04-25T01:21:58.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.587+0000 7fcb15ffb640 1 -- 172.21.15.77:0/3141456067 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcb1c00a850 con 0x7fcb200700c0 2024-04-25T01:21:58.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.587+0000 7fcb15ffb640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcb0406be30 0x7fcb0406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.588+0000 7fcb15ffb640 1 -- 172.21.15.77:0/3141456067 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcb1c08ae90 con 0x7fcb200700c0 2024-04-25T01:21:58.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.588+0000 7fcb24c35640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcb0406be30 0x7fcb0406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.588+0000 7fcb24c35640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcb0406be30 0x7fcb0406e2f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcb10034000 tx=0x7fcb1002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:58.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.592+0000 7fcb15ffb640 1 -- 172.21.15.77:0/3141456067 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcb1c020020 con 0x7fcb200700c0 2024-04-25T01:21:58.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.762+0000 7fcb2769a640 1 -- 172.21.15.77:0/3141456067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcb200b7c70 con 0x7fcb200700c0 2024-04-25T01:21:58.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.763+0000 7fcb15ffb640 1 -- 172.21.15.77:0/3141456067 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7fcb1c019310 con 0x7fcb200700c0 2024-04-25T01:21:58.745 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 -- 172.21.15.77:0/3141456067 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcb0406be30 msgr2=0x7fcb0406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcb0406be30 0x7fcb0406e2f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcb10034000 tx=0x7fcb1002e000 comp rx=0 tx=0).stop 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 -- 172.21.15.77:0/3141456067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb200700c0 msgr2=0x7fcb2014a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb200700c0 0x7fcb2014a0a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00c790 tx=0x7fcb1c00cc50 comp rx=0 tx=0).stop 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 -- 172.21.15.77:0/3141456067 shutdown_connections 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcb0406be30 0x7fcb0406e2f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb200cedb0 0x7fcb200c6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb200cdc50 0x7fcb200c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 --2- 172.21.15.77:0/3141456067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb200700c0 0x7fcb2014a0a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 -- 172.21.15.77:0/3141456067 >> 172.21.15.77:0/3141456067 conn(0x7fcb200d1cf0 msgr2=0x7fcb200b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 -- 172.21.15.77:0/3141456067 shutdown_connections 2024-04-25T01:21:58.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.765+0000 7fcb2769a640 1 -- 172.21.15.77:0/3141456067 wait complete. 2024-04-25T01:21:58.758 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:21:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.875+0000 7f25872ba640 1 Processor -- start 2024-04-25T01:21:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.875+0000 7f25872ba640 1 -- start start 2024-04-25T01:21:58.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.876+0000 7f25872ba640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f258006fcd0 0x7f258015ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.876+0000 7f25872ba640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25800ade10 0x7f258006c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.876+0000 7f25872ba640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f258006cac0 con 0x7f258006c670 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.876+0000 7f25872ba640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f25801589e0 con 0x7f258006fcd0 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.876+0000 7f25872ba640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f25800b3890 con 0x7f25800ade10 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.876+0000 7f2585857640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f258006c670 0x7f2580161ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47486/0 (socket says 172.21.15.77:47486) 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.876+0000 7f2585857640 1 -- 172.21.15.77:0/213536128 learned_addr learned my addr 172.21.15.77:0/213536128 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.876+0000 7f2584855640 1 --2- 172.21.15.77:0/213536128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25800ade10 0x7f258006c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.876+0000 7f2585056640 1 --2- 172.21.15.77:0/213536128 >> v2:172.21.15.121:3300/0 conn(0x7f258006fcd0 0x7f258015ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.876+0000 7f2577fff640 1 -- 172.21.15.77:0/213536128 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059914575 0 0) 0x7f258006cac0 con 0x7f258006c670 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.876+0000 7f2577fff640 1 -- 172.21.15.77:0/213536128 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f256c003520 con 0x7f258006c670 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.877+0000 7f2585056640 1 -- 172.21.15.77:0/213536128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25800ade10 msgr2=0x7f258006c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.877+0000 7f2585056640 1 --2- 172.21.15.77:0/213536128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25800ade10 0x7f258006c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.877+0000 7f2585056640 1 -- 172.21.15.77:0/213536128 >> v1:172.21.15.77:6789/0 conn(0x7f258006c670 legacy=0x7f2580161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.877+0000 7f2585056640 1 -- 172.21.15.77:0/213536128 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25800ae290 con 0x7f258006fcd0 2024-04-25T01:21:58.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.877+0000 7f2584855640 1 --2- 172.21.15.77:0/213536128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25800ade10 0x7f258006c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:21:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.877+0000 7f2585056640 1 --2- 172.21.15.77:0/213536128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f258006fcd0 0x7f258015ef50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f257c009d90 tx=0x7f257c02a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c520ce90c53301f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.877+0000 7f2577fff640 1 -- 172.21.15.77:0/213536128 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c031600 con 0x7f258006fcd0 2024-04-25T01:21:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.878+0000 7f2577fff640 1 -- 172.21.15.77:0/213536128 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f257c0317a0 con 0x7f258006fcd0 2024-04-25T01:21:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.878+0000 7f2577fff640 1 -- 172.21.15.77:0/213536128 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c031b50 con 0x7f258006fcd0 2024-04-25T01:21:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.878+0000 7f25872ba640 1 -- 172.21.15.77:0/213536128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f258006fcd0 msgr2=0x7f258015ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.878+0000 7f25872ba640 1 --2- 172.21.15.77:0/213536128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f258006fcd0 0x7f258015ef50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f257c009d90 tx=0x7f257c02a5d0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.878+0000 7f25872ba640 1 -- 172.21.15.77:0/213536128 shutdown_connections 2024-04-25T01:21:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.878+0000 7f25872ba640 1 --2- 172.21.15.77:0/213536128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25800ade10 0x7f258006c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.878+0000 7f25872ba640 1 --2- 172.21.15.77:0/213536128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f258006fcd0 0x7f258015ef50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.878+0000 7f25872ba640 1 -- 172.21.15.77:0/213536128 >> 172.21.15.77:0/213536128 conn(0x7f2580071550 msgr2=0x7f2580071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.878+0000 7f25872ba640 1 -- 172.21.15.77:0/213536128 shutdown_connections 2024-04-25T01:21:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.878+0000 7f25872ba640 1 -- 172.21.15.77:0/213536128 wait complete. 2024-04-25T01:21:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.879+0000 7f25872ba640 1 Processor -- start 2024-04-25T01:21:58.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.879+0000 7f25872ba640 1 -- start start 2024-04-25T01:21:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f25872ba640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f258006c670 0x7f25801f4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f25872ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f258006fcd0 0x7f2580153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f25872ba640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25800ade10 0x7f258015b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f25872ba640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25800ae7f0 con 0x7f258006fcd0 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f25872ba640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f25800b3560 con 0x7f25800ade10 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f25872ba640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f25800aead0 con 0x7f258006c670 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f2584855640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f258006fcd0 0x7f2580153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f2584855640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f258006fcd0 0x7f2580153f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32806/0 (socket says 172.21.15.77:32806) 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f2584855640 1 -- 172.21.15.77:0/1448647342 learned_addr learned my addr 172.21.15.77:0/1448647342 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f2585056640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f258006c670 0x7f25801f4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f2585857640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25800ade10 0x7f258015b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f2584855640 1 -- 172.21.15.77:0/1448647342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f258006c670 msgr2=0x7f25801f4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f2584855640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f258006c670 0x7f25801f4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f2584855640 1 -- 172.21.15.77:0/1448647342 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25800ade10 msgr2=0x7f258015b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f2584855640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25800ade10 0x7f258015b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.880+0000 7f2584855640 1 -- 172.21.15.77:0/1448647342 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25801f4a90 con 0x7f258006fcd0 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.881+0000 7f2584855640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f258006fcd0 0x7f2580153f90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f257000e990 tx=0x7f257000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.881+0000 7f2575ffb640 1 -- 172.21.15.77:0/1448647342 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2570017070 con 0x7f258006fcd0 2024-04-25T01:21:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.881+0000 7f2575ffb640 1 -- 172.21.15.77:0/1448647342 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f257000c9e0 con 0x7f258006fcd0 2024-04-25T01:21:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.881+0000 7f25872ba640 1 -- 172.21.15.77:0/1448647342 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25801544d0 con 0x7f258006fcd0 2024-04-25T01:21:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.881+0000 7f2575ffb640 1 -- 172.21.15.77:0/1448647342 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257000cdb0 con 0x7f258006fcd0 2024-04-25T01:21:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.881+0000 7f25872ba640 1 -- 172.21.15.77:0/1448647342 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25801531d0 con 0x7f258006fcd0 2024-04-25T01:21:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.882+0000 7f25872ba640 1 -- 172.21.15.77:0/1448647342 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2580153c70 con 0x7f258006fcd0 2024-04-25T01:21:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.882+0000 7f2575ffb640 1 -- 172.21.15.77:0/1448647342 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f257001a070 con 0x7f258006fcd0 2024-04-25T01:21:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.883+0000 7f2575ffb640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f256406be30 0x7f256406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.883+0000 7f2575ffb640 1 -- 172.21.15.77:0/1448647342 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f257008b120 con 0x7f258006fcd0 2024-04-25T01:21:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.883+0000 7f2585056640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f256406be30 0x7f256406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:58.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.883+0000 7f2585056640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f256406be30 0x7f256406e2f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f257c034000 tx=0x7f257c02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:58.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:58.887+0000 7f2575ffb640 1 -- 172.21.15.77:0/1448647342 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2580153c70 con 0x7f258006fcd0 2024-04-25T01:21:59.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.073+0000 7f25872ba640 1 -- 172.21.15.77:0/1448647342 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f25800b5420 con 0x7f258006fcd0 2024-04-25T01:21:59.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.073+0000 7f2575ffb640 1 -- 172.21.15.77:0/1448647342 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f25801f4a90 con 0x7f258006fcd0 2024-04-25T01:21:59.055 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-04-25T01:21:59.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.075+0000 7f25872ba640 1 -- 172.21.15.77:0/1448647342 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f256406be30 msgr2=0x7f256406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:59.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.075+0000 7f25872ba640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f256406be30 0x7f256406e2f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f257c034000 tx=0x7f257c02f000 comp rx=0 tx=0).stop 2024-04-25T01:21:59.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.075+0000 7f25872ba640 1 -- 172.21.15.77:0/1448647342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f258006fcd0 msgr2=0x7f2580153f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:59.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.075+0000 7f25872ba640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f258006fcd0 0x7f2580153f90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f257000e990 tx=0x7f257000ee50 comp rx=0 tx=0).stop 2024-04-25T01:21:59.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.075+0000 7f25872ba640 1 -- 172.21.15.77:0/1448647342 shutdown_connections 2024-04-25T01:21:59.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.075+0000 7f25872ba640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f256406be30 0x7f256406e2f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:59.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.075+0000 7f25872ba640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f258006fcd0 0x7f2580153f90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:59.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.075+0000 7f25872ba640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25800ade10 0x7f258015b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:59.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.075+0000 7f25872ba640 1 --2- 172.21.15.77:0/1448647342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f258006c670 0x7f25801f4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:59.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.075+0000 7f25872ba640 1 -- 172.21.15.77:0/1448647342 >> 172.21.15.77:0/1448647342 conn(0x7f2580071550 msgr2=0x7f25800b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:59.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.076+0000 7f25872ba640 1 -- 172.21.15.77:0/1448647342 shutdown_connections 2024-04-25T01:21:59.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:21:59.076+0000 7f25872ba640 1 -- 172.21.15.77:0/1448647342 wait complete. 2024-04-25T01:21:59.068 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-04-25T01:21:59.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:21:59.068 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:21:59.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi077.stderr:ceph-fuse[41981]: fuse finished with error 0 and tester_r 0 2024-04-25T01:21:59.094 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:22:05.195 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:22:05.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:05.195 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-25T01:22:05.225 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:05.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.391+0000 7f92dd22a640 1 Processor -- start 2024-04-25T01:22:05.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.391+0000 7f92dd22a640 1 -- start start 2024-04-25T01:22:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.392+0000 7f92dd22a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f92d8154ac0 0x7f92d815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.392+0000 7f92dd22a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 0x7f92d806e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.392+0000 7f92dd22a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92d8154ea0 con 0x7f92d80cf2a0 2024-04-25T01:22:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.392+0000 7f92dd22a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f92d806d540 con 0x7f92d8154ac0 2024-04-25T01:22:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.392+0000 7f92dd22a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f92d806eec0 con 0x7f92d806e550 2024-04-25T01:22:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.393+0000 7f92d6d9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f92d8154ac0 0x7f92d815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.393+0000 7f92d659c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f92d80cf2a0 0x7f92d80711f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:44816/0 (socket says 172.21.15.77:44816) 2024-04-25T01:22:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.393+0000 7f92d659c640 1 -- 172.21.15.77:0/2109230732 learned_addr learned my addr 172.21.15.77:0/2109230732 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.393+0000 7f92d759e640 1 --2- 172.21.15.77:0/2109230732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 0x7f92d806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.393+0000 7f92d5d9b640 1 -- 172.21.15.77:0/2109230732 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 164748757 0 0) 0x7f92d8154ea0 con 0x7f92d80cf2a0 2024-04-25T01:22:05.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.393+0000 7f92d5d9b640 1 -- 172.21.15.77:0/2109230732 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92c4003440 con 0x7f92d80cf2a0 2024-04-25T01:22:05.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.393+0000 7f92d6d9d640 1 -- 172.21.15.77:0/2109230732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 msgr2=0x7f92d806e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:05.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.393+0000 7f92d6d9d640 1 --2- 172.21.15.77:0/2109230732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 0x7f92d806e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.393+0000 7f92d6d9d640 1 -- 172.21.15.77:0/2109230732 >> v1:172.21.15.121:6789/0 conn(0x7f92d80cf2a0 legacy=0x7f92d80711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.393+0000 7f92d6d9d640 1 -- 172.21.15.77:0/2109230732 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d806d790 con 0x7f92d8154ac0 2024-04-25T01:22:05.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.393+0000 7f92d759e640 1 --2- 172.21.15.77:0/2109230732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 0x7f92d806e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:05.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.394+0000 7f92d6d9d640 1 --2- 172.21.15.77:0/2109230732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92d8154ac0 0x7f92d815b5f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009d90 tx=0x7f92d002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd6d870d8dc8a7ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:05.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.394+0000 7f92d5d9b640 1 -- 172.21.15.77:0/2109230732 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d0007740 con 0x7f92d8154ac0 2024-04-25T01:22:05.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.394+0000 7f92d5d9b640 1 -- 172.21.15.77:0/2109230732 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92d00078e0 con 0x7f92d8154ac0 2024-04-25T01:22:05.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.394+0000 7f92d5d9b640 1 -- 172.21.15.77:0/2109230732 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d0007c90 con 0x7f92d8154ac0 2024-04-25T01:22:05.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.394+0000 7f92dd22a640 1 -- 172.21.15.77:0/2109230732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92d8154ac0 msgr2=0x7f92d815b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.394+0000 7f92dd22a640 1 --2- 172.21.15.77:0/2109230732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92d8154ac0 0x7f92d815b5f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009d90 tx=0x7f92d002a660 comp rx=0 tx=0).stop 2024-04-25T01:22:05.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.394+0000 7f92dd22a640 1 -- 172.21.15.77:0/2109230732 shutdown_connections 2024-04-25T01:22:05.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.394+0000 7f92dd22a640 1 --2- 172.21.15.77:0/2109230732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 0x7f92d806e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.394+0000 7f92dd22a640 1 --2- 172.21.15.77:0/2109230732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92d8154ac0 0x7f92d815b5f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.394+0000 7f92dd22a640 1 -- 172.21.15.77:0/2109230732 >> 172.21.15.77:0/2109230732 conn(0x7f92d80b1820 msgr2=0x7f92d80b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:05.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.395+0000 7f92dd22a640 1 -- 172.21.15.77:0/2109230732 shutdown_connections 2024-04-25T01:22:05.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.395+0000 7f92dd22a640 1 -- 172.21.15.77:0/2109230732 wait complete. 2024-04-25T01:22:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.396+0000 7f92dd22a640 1 Processor -- start 2024-04-25T01:22:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.396+0000 7f92dd22a640 1 -- start start 2024-04-25T01:22:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92dd22a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 0x7f92d81ef980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92dd22a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92d80cf2a0 0x7f92d8157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92dd22a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92d8154ac0 0x7f92d814df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92dd22a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f92d806e030 con 0x7f92d8154ac0 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92dd22a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f92d806dd50 con 0x7f92d80cf2a0 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92dd22a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f92d8071900 con 0x7f92d806e550 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92d759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92d8154ac0 0x7f92d814df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92d759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92d8154ac0 0x7f92d814df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32816/0 (socket says 172.21.15.77:32816) 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92d759e640 1 -- 172.21.15.77:0/3322776812 learned_addr learned my addr 172.21.15.77:0/3322776812 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92d6d9d640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 0x7f92d81ef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92d659c640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92d80cf2a0 0x7f92d8157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92d6d9d640 1 -- 172.21.15.77:0/3322776812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92d80cf2a0 msgr2=0x7f92d8157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92d6d9d640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92d80cf2a0 0x7f92d8157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92d6d9d640 1 -- 172.21.15.77:0/3322776812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92d8154ac0 msgr2=0x7f92d814df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92d6d9d640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92d8154ac0 0x7f92d814df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92d6d9d640 1 -- 172.21.15.77:0/3322776812 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d81581e0 con 0x7f92d806e550 2024-04-25T01:22:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.397+0000 7f92d759e640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92d8154ac0 0x7f92d814df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.398+0000 7f92d6d9d640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 0x7f92d81ef980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f92d0031000 tx=0x7f92d0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.398+0000 7f92cf7fe640 1 -- 172.21.15.77:0/3322776812 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d003f070 con 0x7f92d806e550 2024-04-25T01:22:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.398+0000 7f92cf7fe640 1 -- 172.21.15.77:0/3322776812 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92d0031c40 con 0x7f92d806e550 2024-04-25T01:22:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.398+0000 7f92dd22a640 1 -- 172.21.15.77:0/3322776812 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92d81f0110 con 0x7f92d806e550 2024-04-25T01:22:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.398+0000 7f92cf7fe640 1 -- 172.21.15.77:0/3322776812 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d0009450 con 0x7f92d806e550 2024-04-25T01:22:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.398+0000 7f92dd22a640 1 -- 172.21.15.77:0/3322776812 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92d81f0bf0 con 0x7f92d806e550 2024-04-25T01:22:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.399+0000 7f92dd22a640 1 -- 172.21.15.77:0/3322776812 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92d8068fb0 con 0x7f92d806e550 2024-04-25T01:22:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.399+0000 7f92cf7fe640 1 -- 172.21.15.77:0/3322776812 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f92d0044460 con 0x7f92d806e550 2024-04-25T01:22:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.400+0000 7f92cf7fe640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f92b806be30 0x7f92b806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.400+0000 7f92cf7fe640 1 -- 172.21.15.77:0/3322776812 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f92d00ac750 con 0x7f92d806e550 2024-04-25T01:22:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.400+0000 7f92d659c640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f92b806be30 0x7f92b806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.400+0000 7f92d659c640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f92b806be30 0x7f92b806e2f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f92d8157510 tx=0x7f92c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:05.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.405+0000 7f92cf7fe640 1 -- 172.21.15.77:0/3322776812 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f92d007a770 con 0x7f92d806e550 2024-04-25T01:22:05.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.578+0000 7f92dd22a640 1 -- 172.21.15.77:0/3322776812 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92d80b4000 con 0x7f92d806e550 2024-04-25T01:22:05.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.580+0000 7f92cf7fe640 1 -- 172.21.15.77:0/3322776812 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f92d007ea70 con 0x7f92d806e550 2024-04-25T01:22:05.562 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:05.562 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 -- 172.21.15.77:0/3322776812 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f92b806be30 msgr2=0x7f92b806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f92b806be30 0x7f92b806e2f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f92d8157510 tx=0x7f92c0009040 comp rx=0 tx=0).stop 2024-04-25T01:22:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 -- 172.21.15.77:0/3322776812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 msgr2=0x7f92d81ef980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 0x7f92d81ef980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f92d0031000 tx=0x7f92d0004a40 comp rx=0 tx=0).stop 2024-04-25T01:22:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 -- 172.21.15.77:0/3322776812 shutdown_connections 2024-04-25T01:22:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f92b806be30 0x7f92b806e2f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92d8154ac0 0x7f92d814df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92d80cf2a0 0x7f92d8157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 --2- 172.21.15.77:0/3322776812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92d806e550 0x7f92d81ef980 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 -- 172.21.15.77:0/3322776812 >> 172.21.15.77:0/3322776812 conn(0x7f92d80b1820 msgr2=0x7f92d815df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:05.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 -- 172.21.15.77:0/3322776812 shutdown_connections 2024-04-25T01:22:05.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.582+0000 7f92dd22a640 1 -- 172.21.15.77:0/3322776812 wait complete. 2024-04-25T01:22:05.575 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:05.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.689+0000 7f5f70de7640 1 Processor -- start 2024-04-25T01:22:05.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.689+0000 7f5f70de7640 1 -- start start 2024-04-25T01:22:05.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f70de7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5f6c0cefe0 0x7f5f6c0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f70de7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f6c06e000 0x7f5f6c163440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f70de7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f6c0b2b20 0x7f5f6c0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f70de7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f5f6c163980 con 0x7f5f6c0b2b20 2024-04-25T01:22:05.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f70de7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5f6c06cf60 con 0x7f5f6c0cefe0 2024-04-25T01:22:05.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f70de7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c06d240 con 0x7f5f6c06e000 2024-04-25T01:22:05.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f6ad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f6c0b2b20 0x7f5f6c0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f6ad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f6c0b2b20 0x7f5f6c0b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32818/0 (socket says 172.21.15.77:32818) 2024-04-25T01:22:05.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f6ad9d640 1 -- 172.21.15.77:0/429673517 learned_addr learned my addr 172.21.15.77:0/429673517 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:05.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f69d9b640 1 --2- 172.21.15.77:0/429673517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f6c06e000 0x7f5f6c163440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f6a59c640 1 --2- 172.21.15.77:0/429673517 >> v2:172.21.15.121:3300/0 conn(0x7f5f6c0cefe0 0x7f5f6c0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f6ad9d640 1 -- 172.21.15.77:0/429673517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f6c06e000 msgr2=0x7f5f6c163440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:05.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f6ad9d640 1 --2- 172.21.15.77:0/429673517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f6c06e000 0x7f5f6c163440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f6ad9d640 1 -- 172.21.15.77:0/429673517 >> v2:172.21.15.121:3300/0 conn(0x7f5f6c0cefe0 msgr2=0x7f5f6c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:05.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f6ad9d640 1 --2- 172.21.15.77:0/429673517 >> v2:172.21.15.121:3300/0 conn(0x7f5f6c0cefe0 0x7f5f6c0b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.690+0000 7f5f6ad9d640 1 -- 172.21.15.77:0/429673517 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c06d520 con 0x7f5f6c0b2b20 2024-04-25T01:22:05.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.691+0000 7f5f6ad9d640 1 --2- 172.21.15.77:0/429673517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f6c0b2b20 0x7f5f6c0b2fb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c0739a0 tx=0x7f5f6402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac82569e900ce3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:05.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.691+0000 7f5f6959a640 1 -- 172.21.15.77:0/429673517 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64037070 con 0x7f5f6c0b2b20 2024-04-25T01:22:05.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.691+0000 7f5f6959a640 1 -- 172.21.15.77:0/429673517 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f64032440 con 0x7f5f6c0b2b20 2024-04-25T01:22:05.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.691+0000 7f5f6959a640 1 -- 172.21.15.77:0/429673517 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64032810 con 0x7f5f6c0b2b20 2024-04-25T01:22:05.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.692+0000 7f5f70de7640 1 -- 172.21.15.77:0/429673517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f6c0b2b20 msgr2=0x7f5f6c0b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.692+0000 7f5f70de7640 1 --2- 172.21.15.77:0/429673517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f6c0b2b20 0x7f5f6c0b2fb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c0739a0 tx=0x7f5f6402a140 comp rx=0 tx=0).stop 2024-04-25T01:22:05.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.692+0000 7f5f70de7640 1 -- 172.21.15.77:0/429673517 shutdown_connections 2024-04-25T01:22:05.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.692+0000 7f5f70de7640 1 --2- 172.21.15.77:0/429673517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f6c0b2b20 0x7f5f6c0b2fb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.692+0000 7f5f70de7640 1 --2- 172.21.15.77:0/429673517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f6c06e000 0x7f5f6c163440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.692+0000 7f5f70de7640 1 --2- 172.21.15.77:0/429673517 >> v2:172.21.15.121:3300/0 conn(0x7f5f6c0cefe0 0x7f5f6c0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.692+0000 7f5f70de7640 1 -- 172.21.15.77:0/429673517 >> 172.21.15.77:0/429673517 conn(0x7f5f6c0b1820 msgr2=0x7f5f6c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:05.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.692+0000 7f5f70de7640 1 -- 172.21.15.77:0/429673517 shutdown_connections 2024-04-25T01:22:05.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.692+0000 7f5f70de7640 1 -- 172.21.15.77:0/429673517 wait complete. 2024-04-25T01:22:05.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.693+0000 7f5f70de7640 1 Processor -- start 2024-04-25T01:22:05.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.693+0000 7f5f70de7640 1 -- start start 2024-04-25T01:22:05.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f70de7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f6c06e000 0x7f5f6c14bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f70de7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c0b2b20 0x7f5f6c0abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f70de7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f6c0cefe0 0x7f5f6c0ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f70de7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c06e490 con 0x7f5f6c06e000 2024-04-25T01:22:05.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f70de7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c06dae0 con 0x7f5f6c0b2b20 2024-04-25T01:22:05.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f70de7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c070c50 con 0x7f5f6c0cefe0 2024-04-25T01:22:05.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f6a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f6c06e000 0x7f5f6c14bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f6a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f6c06e000 0x7f5f6c14bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32820/0 (socket says 172.21.15.77:32820) 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f6a59c640 1 -- 172.21.15.77:0/673425876 learned_addr learned my addr 172.21.15.77:0/673425876 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f69d9b640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c0b2b20 0x7f5f6c0abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f6ad9d640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f6c0cefe0 0x7f5f6c0ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f6a59c640 1 -- 172.21.15.77:0/673425876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f6c0cefe0 msgr2=0x7f5f6c0ac130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f6a59c640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f6c0cefe0 0x7f5f6c0ac130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f6a59c640 1 -- 172.21.15.77:0/673425876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c0b2b20 msgr2=0x7f5f6c0abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f6a59c640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c0b2b20 0x7f5f6c0abbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.694+0000 7f5f6a59c640 1 -- 172.21.15.77:0/673425876 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c14c1b0 con 0x7f5f6c06e000 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.695+0000 7f5f6a59c640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f6c06e000 0x7f5f6c14bc70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c007e40 tx=0x7f5f5c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.695+0000 7f5f5affd640 1 -- 172.21.15.77:0/673425876 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c014620 con 0x7f5f6c06e000 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.695+0000 7f5f5affd640 1 -- 172.21.15.77:0/673425876 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f5c019070 con 0x7f5f6c06e000 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.695+0000 7f5f5affd640 1 -- 172.21.15.77:0/673425876 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c014a00 con 0x7f5f6c06e000 2024-04-25T01:22:05.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.695+0000 7f5f70de7640 1 -- 172.21.15.77:0/673425876 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6c1ee720 con 0x7f5f6c06e000 2024-04-25T01:22:05.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.695+0000 7f5f70de7640 1 -- 172.21.15.77:0/673425876 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6c0a9930 con 0x7f5f6c06e000 2024-04-25T01:22:05.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.696+0000 7f5f70de7640 1 -- 172.21.15.77:0/673425876 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f6c068fb0 con 0x7f5f6c06e000 2024-04-25T01:22:05.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.696+0000 7f5f5affd640 1 -- 172.21.15.77:0/673425876 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5f5c014ba0 con 0x7f5f6c06e000 2024-04-25T01:22:05.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.697+0000 7f5f5affd640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f4c06be30 0x7f5f4c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.697+0000 7f5f69d9b640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f4c06be30 0x7f5f4c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.697+0000 7f5f5affd640 1 -- 172.21.15.77:0/673425876 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5f5c08b8a0 con 0x7f5f6c06e000 2024-04-25T01:22:05.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.697+0000 7f5f69d9b640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f4c06be30 0x7f5f4c06e2f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c14c490 tx=0x7f5f60008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:05.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.701+0000 7f5f5affd640 1 -- 172.21.15.77:0/673425876 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5f5c0598c0 con 0x7f5f6c06e000 2024-04-25T01:22:05.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.875+0000 7f5f70de7640 1 -- 172.21.15.77:0/673425876 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f6c072c30 con 0x7f5f6c06e000 2024-04-25T01:22:05.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.876+0000 7f5f5affd640 1 -- 172.21.15.77:0/673425876 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f5f5c05dbc0 con 0x7f5f6c06e000 2024-04-25T01:22:05.858 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:05.858 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 -- 172.21.15.77:0/673425876 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f4c06be30 msgr2=0x7f5f4c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f4c06be30 0x7f5f4c06e2f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c14c490 tx=0x7f5f60008040 comp rx=0 tx=0).stop 2024-04-25T01:22:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 -- 172.21.15.77:0/673425876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f6c06e000 msgr2=0x7f5f6c14bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f6c06e000 0x7f5f6c14bc70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c007e40 tx=0x7f5f5c00e4b0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 -- 172.21.15.77:0/673425876 shutdown_connections 2024-04-25T01:22:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5f4c06be30 0x7f5f4c06e2f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f6c0cefe0 0x7f5f6c0ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c0b2b20 0x7f5f6c0abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 --2- 172.21.15.77:0/673425876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f6c06e000 0x7f5f6c14bc70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 -- 172.21.15.77:0/673425876 >> 172.21.15.77:0/673425876 conn(0x7f5f6c0b1820 msgr2=0x7f5f6c1667d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 -- 172.21.15.77:0/673425876 shutdown_connections 2024-04-25T01:22:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.878+0000 7f5f70de7640 1 -- 172.21.15.77:0/673425876 wait complete. 2024-04-25T01:22:05.871 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T01:22:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.988+0000 7fc70f35a640 1 Processor -- start 2024-04-25T01:22:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.988+0000 7fc70f35a640 1 -- start start 2024-04-25T01:22:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.988+0000 7fc70f35a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc708158a70 0x7fc7080b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70f35a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc7080b37c0 0x7fc7080b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70f35a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc70815c210 con 0x7fc70815b520 2024-04-25T01:22:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70f35a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fc7080b2ce0 con 0x7fc708158a70 2024-04-25T01:22:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70f35a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc7080b2fc0 con 0x7fc7080b37c0 2024-04-25T01:22:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70d0f6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc708158a70 0x7fc7080b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70d0f6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc708158a70 0x7fc7080b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32832/0 (socket says 172.21.15.77:32832) 2024-04-25T01:22:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70d0f6640 1 -- 172.21.15.77:0/3991224068 learned_addr learned my addr 172.21.15.77:0/3991224068 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70c8f5640 1 --2- 172.21.15.77:0/3991224068 >> v2:172.21.15.121:3300/0 conn(0x7fc7080b37c0 0x7fc7080b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70d0f6640 1 -- 172.21.15.77:0/3991224068 >> v2:172.21.15.121:3300/0 conn(0x7fc7080b37c0 msgr2=0x7fc7080b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70d0f6640 1 --2- 172.21.15.77:0/3991224068 >> v2:172.21.15.121:3300/0 conn(0x7fc7080b37c0 0x7fc7080b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70d0f6640 1 -- 172.21.15.77:0/3991224068 >> v1:172.21.15.121:6789/0 conn(0x7fc70815b520 legacy=0x7fc70815b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.989+0000 7fc70d0f6640 1 -- 172.21.15.77:0/3991224068 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7080b32a0 con 0x7fc708158a70 2024-04-25T01:22:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc70d0f6640 1 --2- 172.21.15.77:0/3991224068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc708158a70 0x7fc7080b40c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc70000bb10 tx=0x7fc70002c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8376069165b6a18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc6fffff640 1 -- 172.21.15.77:0/3991224068 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70002f070 con 0x7fc708158a70 2024-04-25T01:22:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc6fffff640 1 -- 172.21.15.77:0/3991224068 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc700038070 con 0x7fc708158a70 2024-04-25T01:22:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc6fffff640 1 -- 172.21.15.77:0/3991224068 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc700032a00 con 0x7fc708158a70 2024-04-25T01:22:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc70f35a640 1 -- 172.21.15.77:0/3991224068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc708158a70 msgr2=0x7fc7080b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc70f35a640 1 --2- 172.21.15.77:0/3991224068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc708158a70 0x7fc7080b40c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc70000bb10 tx=0x7fc70002c920 comp rx=0 tx=0).stop 2024-04-25T01:22:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc70f35a640 1 -- 172.21.15.77:0/3991224068 shutdown_connections 2024-04-25T01:22:05.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc70f35a640 1 --2- 172.21.15.77:0/3991224068 >> v2:172.21.15.121:3300/0 conn(0x7fc7080b37c0 0x7fc7080b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc70f35a640 1 --2- 172.21.15.77:0/3991224068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc708158a70 0x7fc7080b40c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc70f35a640 1 -- 172.21.15.77:0/3991224068 >> 172.21.15.77:0/3991224068 conn(0x7fc70806b720 msgr2=0x7fc7080b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:05.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc70f35a640 1 -- 172.21.15.77:0/3991224068 shutdown_connections 2024-04-25T01:22:05.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.990+0000 7fc70f35a640 1 -- 172.21.15.77:0/3991224068 wait complete. 2024-04-25T01:22:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.991+0000 7fc70f35a640 1 Processor -- start 2024-04-25T01:22:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.992+0000 7fc70f35a640 1 -- start start 2024-04-25T01:22:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.992+0000 7fc70f35a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7080b37c0 0x7fc708157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.992+0000 7fc70f35a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc708158a70 0x7fc7080a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.992+0000 7fc70f35a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70815b520 0x7fc7080b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.992+0000 7fc70f35a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc70806cb70 con 0x7fc70815b520 2024-04-25T01:22:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.992+0000 7fc70f35a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc70806c9f0 con 0x7fc708158a70 2024-04-25T01:22:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.992+0000 7fc70f35a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc70806ccf0 con 0x7fc7080b37c0 2024-04-25T01:22:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.992+0000 7fc70d8f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70815b520 0x7fc7080b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.992+0000 7fc70d8f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70815b520 0x7fc7080b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32834/0 (socket says 172.21.15.77:32834) 2024-04-25T01:22:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.992+0000 7fc70d8f7640 1 -- 172.21.15.77:0/446919225 learned_addr learned my addr 172.21.15.77:0/446919225 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc70c8f5640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc708158a70 0x7fc7080a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc70d0f6640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7080b37c0 0x7fc708157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc70d8f7640 1 -- 172.21.15.77:0/446919225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7080b37c0 msgr2=0x7fc708157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc70d8f7640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7080b37c0 0x7fc708157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc70d8f7640 1 -- 172.21.15.77:0/446919225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc708158a70 msgr2=0x7fc7080a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc70d8f7640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc708158a70 0x7fc7080a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc70d8f7640 1 -- 172.21.15.77:0/446919225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7080aa8c0 con 0x7fc70815b520 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc70d8f7640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70815b520 0x7fc7080b0b40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc70400b990 tx=0x7fc70400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc6fdffb640 1 -- 172.21.15.77:0/446919225 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704019070 con 0x7fc70815b520 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc6fdffb640 1 -- 172.21.15.77:0/446919225 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc70400cba0 con 0x7fc70815b520 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc70f35a640 1 -- 172.21.15.77:0/446919225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7080aaba0 con 0x7fc70815b520 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc6fdffb640 1 -- 172.21.15.77:0/446919225 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704014440 con 0x7fc70815b520 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.993+0000 7fc70f35a640 1 -- 172.21.15.77:0/446919225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7080ab160 con 0x7fc70815b520 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.994+0000 7fc70f35a640 1 -- 172.21.15.77:0/446919225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7080acaf0 con 0x7fc70815b520 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.994+0000 7fc6fdffb640 1 -- 172.21.15.77:0/446919225 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc70401c070 con 0x7fc70815b520 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.995+0000 7fc6fdffb640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc6ec06be30 0x7fc6ec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.995+0000 7fc70d0f6640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc6ec06be30 0x7fc6ec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.995+0000 7fc6fdffb640 1 -- 172.21.15.77:0/446919225 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc70408af40 con 0x7fc70815b520 2024-04-25T01:22:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.995+0000 7fc70d0f6640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc6ec06be30 0x7fc6ec06e2f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc700037000 tx=0x7fc700032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:05.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:05.999+0000 7fc6fdffb640 1 -- 172.21.15.77:0/446919225 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc7080acaf0 con 0x7fc70815b520 2024-04-25T01:22:06.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.215+0000 7fc70f35a640 1 -- 172.21.15.77:0/446919225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc7080b3c70 con 0x7fc70815b520 2024-04-25T01:22:06.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.220+0000 7fc6fdffb640 1 -- 172.21.15.77:0/446919225 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7fc7040595b0 con 0x7fc70815b520 2024-04-25T01:22:06.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.222+0000 7fc70f35a640 1 -- 172.21.15.77:0/446919225 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc6ec06be30 msgr2=0x7fc6ec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:06.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.222+0000 7fc70f35a640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc6ec06be30 0x7fc6ec06e2f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc700037000 tx=0x7fc700032000 comp rx=0 tx=0).stop 2024-04-25T01:22:06.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.222+0000 7fc70f35a640 1 -- 172.21.15.77:0/446919225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70815b520 msgr2=0x7fc7080b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:06.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.222+0000 7fc70f35a640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70815b520 0x7fc7080b0b40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc70400b990 tx=0x7fc70400be50 comp rx=0 tx=0).stop 2024-04-25T01:22:06.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.222+0000 7fc70f35a640 1 -- 172.21.15.77:0/446919225 shutdown_connections 2024-04-25T01:22:06.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.222+0000 7fc70f35a640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc6ec06be30 0x7fc6ec06e2f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:06.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.222+0000 7fc70f35a640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70815b520 0x7fc7080b0b40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:06.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.222+0000 7fc70f35a640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc708158a70 0x7fc7080a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:06.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.222+0000 7fc70f35a640 1 --2- 172.21.15.77:0/446919225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7080b37c0 0x7fc708157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:06.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.222+0000 7fc70f35a640 1 -- 172.21.15.77:0/446919225 >> 172.21.15.77:0/446919225 conn(0x7fc70806b720 msgr2=0x7fc708070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:06.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.222+0000 7fc70f35a640 1 -- 172.21.15.77:0/446919225 shutdown_connections 2024-04-25T01:22:06.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.223+0000 7fc70f35a640 1 -- 172.21.15.77:0/446919225 wait complete. 2024-04-25T01:22:06.215 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-25T01:22:06.215 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T01:22:06.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.332+0000 7f4105589640 1 Processor -- start 2024-04-25T01:22:06.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.332+0000 7f4105589640 1 -- start start 2024-04-25T01:22:06.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f4105589640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41000b9a90 0x7f41000d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:06.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f4105589640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f41000d27b0 0x7f41000d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:06.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f4105589640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41000cea40 con 0x7f41000d23d0 2024-04-25T01:22:06.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f4105589640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f41000cece0 con 0x7f41000d27b0 2024-04-25T01:22:06.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f4105589640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f41000cefc0 con 0x7f41000b9a90 2024-04-25T01:22:06.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f40ff7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f41000d27b0 0x7f41000d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:06.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f40ff7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f41000d27b0 0x7f41000d0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32836/0 (socket says 172.21.15.77:32836) 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f40ff7fe640 1 -- 172.21.15.77:0/1039633932 learned_addr learned my addr 172.21.15.77:0/1039633932 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f40fe7fc640 1 --2- 172.21.15.77:0/1039633932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41000b9a90 0x7f41000d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f40ff7fe640 1 -- 172.21.15.77:0/1039633932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41000b9a90 msgr2=0x7f41000d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f40ff7fe640 1 --2- 172.21.15.77:0/1039633932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41000b9a90 0x7f41000d4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f40ff7fe640 1 -- 172.21.15.77:0/1039633932 >> v1:172.21.15.121:6789/0 conn(0x7f41000d23d0 legacy=0x7f41000b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.333+0000 7f40ff7fe640 1 -- 172.21.15.77:0/1039633932 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41000cf2a0 con 0x7f41000d27b0 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f40ff7fe640 1 --2- 172.21.15.77:0/1039633932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41000d27b0 0x7f41000d0ee0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f40f8009e50 tx=0x7f40f802a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5e1648cc9b2316 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f40fdffb640 1 -- 172.21.15.77:0/1039633932 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f8036070 con 0x7f41000d27b0 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f40fdffb640 1 -- 172.21.15.77:0/1039633932 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40f8031600 con 0x7f41000d27b0 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f40fdffb640 1 -- 172.21.15.77:0/1039633932 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f80319d0 con 0x7f41000d27b0 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f4105589640 1 -- 172.21.15.77:0/1039633932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41000d27b0 msgr2=0x7f41000d0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f4105589640 1 --2- 172.21.15.77:0/1039633932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41000d27b0 0x7f41000d0ee0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f40f8009e50 tx=0x7f40f802a750 comp rx=0 tx=0).stop 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f4105589640 1 -- 172.21.15.77:0/1039633932 shutdown_connections 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f4105589640 1 --2- 172.21.15.77:0/1039633932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41000d27b0 0x7f41000d0ee0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f4105589640 1 --2- 172.21.15.77:0/1039633932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41000b9a90 0x7f41000d4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:06.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f4105589640 1 -- 172.21.15.77:0/1039633932 >> 172.21.15.77:0/1039633932 conn(0x7f41000d3e30 msgr2=0x7f41000d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:06.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f4105589640 1 -- 172.21.15.77:0/1039633932 shutdown_connections 2024-04-25T01:22:06.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.334+0000 7f4105589640 1 -- 172.21.15.77:0/1039633932 wait complete. 2024-04-25T01:22:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.336+0000 7f4105589640 1 Processor -- start 2024-04-25T01:22:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.336+0000 7f4105589640 1 -- start start 2024-04-25T01:22:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.336+0000 7f4105589640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41000b9a90 0x7f4100152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.336+0000 7f4105589640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41000d23d0 0x7f41000c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.336+0000 7f4105589640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41000d27b0 0x7f410006bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.336+0000 7f4105589640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41000cfd80 con 0x7f41000d23d0 2024-04-25T01:22:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.336+0000 7f4105589640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f41000cf860 con 0x7f41000d27b0 2024-04-25T01:22:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.336+0000 7f4105589640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f41000b6c80 con 0x7f41000b9a90 2024-04-25T01:22:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.336+0000 7f40fe7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41000d23d0 0x7f41000c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40fe7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41000d23d0 0x7f41000c7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32844/0 (socket says 172.21.15.77:32844) 2024-04-25T01:22:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40fe7fc640 1 -- 172.21.15.77:0/2452899493 learned_addr learned my addr 172.21.15.77:0/2452899493 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40feffd640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41000b9a90 0x7f4100152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40ff7fe640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41000d27b0 0x7f410006bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40fe7fc640 1 -- 172.21.15.77:0/2452899493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41000b9a90 msgr2=0x7f4100152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40fe7fc640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41000b9a90 0x7f4100152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40fe7fc640 1 -- 172.21.15.77:0/2452899493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41000d27b0 msgr2=0x7f410006bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40fe7fc640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41000d27b0 0x7f410006bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40fe7fc640 1 -- 172.21.15.77:0/2452899493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41000b1340 con 0x7f41000d23d0 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40fe7fc640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41000d23d0 0x7f41000c7960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f40f400d800 tx=0x7f40f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40e77fe640 1 -- 172.21.15.77:0/2452899493 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f400b6c0 con 0x7f41000d23d0 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40e77fe640 1 -- 172.21.15.77:0/2452899493 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40f4019070 con 0x7f41000d23d0 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f4105589640 1 -- 172.21.15.77:0/2452899493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41000c83b0 con 0x7f41000d23d0 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f40e77fe640 1 -- 172.21.15.77:0/2452899493 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f400ba80 con 0x7f41000d23d0 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.337+0000 7f4105589640 1 -- 172.21.15.77:0/2452899493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41000c8970 con 0x7f41000d23d0 2024-04-25T01:22:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.338+0000 7f4105589640 1 -- 172.21.15.77:0/2452899493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f410006bad0 con 0x7f41000d23d0 2024-04-25T01:22:06.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.339+0000 7f40e77fe640 1 -- 172.21.15.77:0/2452899493 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f40f400bc20 con 0x7f41000d23d0 2024-04-25T01:22:06.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.339+0000 7f40e77fe640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f40ec06be60 0x7f40ec06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:06.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.339+0000 7f40e77fe640 1 -- 172.21.15.77:0/2452899493 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f40f408b460 con 0x7f41000d23d0 2024-04-25T01:22:06.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.339+0000 7f40feffd640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f40ec06be60 0x7f40ec06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:06.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.339+0000 7f40feffd640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f40ec06be60 0x7f40ec06e320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f41001611f0 tx=0x7f40f0007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:06.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.343+0000 7f40e77fe640 1 -- 172.21.15.77:0/2452899493 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f410006bad0 con 0x7f41000d23d0 2024-04-25T01:22:06.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:06.552+0000 7f4105589640 1 -- 172.21.15.77:0/2452899493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f410008df50 con 0x7f41000d23d0 2024-04-25T01:22:07.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.225+0000 7f40e77fe640 1 -- 172.21.15.77:0/2452899493 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7f40f4059510 con 0x7f41000d23d0 2024-04-25T01:22:07.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 -- 172.21.15.77:0/2452899493 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f40ec06be60 msgr2=0x7f40ec06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f40ec06be60 0x7f40ec06e320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f41001611f0 tx=0x7f40f0007e30 comp rx=0 tx=0).stop 2024-04-25T01:22:07.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 -- 172.21.15.77:0/2452899493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41000d23d0 msgr2=0x7f41000c7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41000d23d0 0x7f41000c7960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f40f400d800 tx=0x7f40f400dcc0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 -- 172.21.15.77:0/2452899493 shutdown_connections 2024-04-25T01:22:07.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f40ec06be60 0x7f40ec06e320 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41000d23d0 0x7f41000c7960 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41000d27b0 0x7f410006bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 --2- 172.21.15.77:0/2452899493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41000b9a90 0x7f4100152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 -- 172.21.15.77:0/2452899493 >> 172.21.15.77:0/2452899493 conn(0x7f41000d3e30 msgr2=0x7f41000ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:07.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 -- 172.21.15.77:0/2452899493 shutdown_connections 2024-04-25T01:22:07.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.228+0000 7f4105589640 1 -- 172.21.15.77:0/2452899493 wait complete. 2024-04-25T01:22:07.223 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T01:22:07.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:07.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.343+0000 7fd923b7e640 1 Processor -- start 2024-04-25T01:22:07.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.344+0000 7fd923b7e640 1 -- start start 2024-04-25T01:22:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.344+0000 7fd923b7e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd91c06fcd0 0x7fd91c15ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.344+0000 7fd923b7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06c840 0x7fd91c0b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.344+0000 7fd923b7e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd91c0b3560 con 0x7fd91c0ade10 2024-04-25T01:22:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.344+0000 7fd923b7e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd91c1589e0 con 0x7fd91c06fcd0 2024-04-25T01:22:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.344+0000 7fd923b7e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd91c0b3890 con 0x7fd91c06c840 2024-04-25T01:22:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.345+0000 7fd92191a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd91c06fcd0 0x7fd91c15ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.345+0000 7fd92211b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06c840 0x7fd91c0b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.345+0000 7fd92191a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd91c06fcd0 0x7fd91c15ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48526/0 (socket says 172.21.15.77:48526) 2024-04-25T01:22:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.345+0000 7fd921119640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd91c0ade10 0x7fd91c06c130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:59560/0 (socket says 172.21.15.77:59560) 2024-04-25T01:22:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.345+0000 7fd92211b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06c840 0x7fd91c0b3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33550/0 (socket says 172.21.15.77:33550) 2024-04-25T01:22:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.345+0000 7fd92191a640 1 -- 172.21.15.77:0/1610434291 learned_addr learned my addr 172.21.15.77:0/1610434291 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:07.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.345+0000 7fd92211b640 1 -- 172.21.15.77:0/1610434291 >> v2:172.21.15.121:3300/0 conn(0x7fd91c06fcd0 msgr2=0x7fd91c15ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.345+0000 7fd92211b640 1 --2- 172.21.15.77:0/1610434291 >> v2:172.21.15.121:3300/0 conn(0x7fd91c06fcd0 0x7fd91c15ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.345+0000 7fd92211b640 1 -- 172.21.15.77:0/1610434291 >> v1:172.21.15.121:6789/0 conn(0x7fd91c0ade10 legacy=0x7fd91c06c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.345+0000 7fd92211b640 1 -- 172.21.15.77:0/1610434291 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91c0ae510 con 0x7fd91c06c840 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.346+0000 7fd92211b640 1 --2- 172.21.15.77:0/1610434291 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06c840 0x7fd91c0b3d20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd914009d90 tx=0x7fd91402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59e33a9a42ad62f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.346+0000 7fd920918640 1 -- 172.21.15.77:0/1610434291 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd914031600 con 0x7fd91c06c840 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.346+0000 7fd920918640 1 -- 172.21.15.77:0/1610434291 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9140317a0 con 0x7fd91c06c840 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.346+0000 7fd920918640 1 -- 172.21.15.77:0/1610434291 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd914031b70 con 0x7fd91c06c840 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.347+0000 7fd923b7e640 1 -- 172.21.15.77:0/1610434291 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06c840 msgr2=0x7fd91c0b3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.347+0000 7fd923b7e640 1 --2- 172.21.15.77:0/1610434291 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06c840 0x7fd91c0b3d20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd914009d90 tx=0x7fd91402a5d0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.347+0000 7fd923b7e640 1 -- 172.21.15.77:0/1610434291 shutdown_connections 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.347+0000 7fd923b7e640 1 --2- 172.21.15.77:0/1610434291 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06c840 0x7fd91c0b3d20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.347+0000 7fd923b7e640 1 --2- 172.21.15.77:0/1610434291 >> v2:172.21.15.121:3300/0 conn(0x7fd91c06fcd0 0x7fd91c15ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.347+0000 7fd923b7e640 1 -- 172.21.15.77:0/1610434291 >> 172.21.15.77:0/1610434291 conn(0x7fd91c071550 msgr2=0x7fd91c071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:07.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.347+0000 7fd923b7e640 1 -- 172.21.15.77:0/1610434291 shutdown_connections 2024-04-25T01:22:07.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.347+0000 7fd923b7e640 1 -- 172.21.15.77:0/1610434291 wait complete. 2024-04-25T01:22:07.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.348+0000 7fd923b7e640 1 Processor -- start 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.348+0000 7fd923b7e640 1 -- start start 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.348+0000 7fd923b7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c06c840 0x7fd91c15d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.348+0000 7fd923b7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06fcd0 0x7fd91c1f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.348+0000 7fd923b7e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0ade10 0x7fd91c0cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.348+0000 7fd923b7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c0aead0 con 0x7fd91c06c840 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.348+0000 7fd923b7e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c0ae2c0 con 0x7fd91c0ade10 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.348+0000 7fd923b7e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd91c0b4260 con 0x7fd91c06fcd0 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd92191a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c06c840 0x7fd91c15d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd92191a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c06c840 0x7fd91c15d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55990/0 (socket says 172.21.15.77:55990) 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd92191a640 1 -- 172.21.15.77:0/1699417567 learned_addr learned my addr 172.21.15.77:0/1699417567 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd92211b640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0ade10 0x7fd91c0cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd921119640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06fcd0 0x7fd91c1f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd92191a640 1 -- 172.21.15.77:0/1699417567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06fcd0 msgr2=0x7fd91c1f4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd92191a640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06fcd0 0x7fd91c1f4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd92191a640 1 -- 172.21.15.77:0/1699417567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0ade10 msgr2=0x7fd91c0cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd92191a640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0ade10 0x7fd91c0cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd92191a640 1 -- 172.21.15.77:0/1699417567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91c1f4f90 con 0x7fd91c06c840 2024-04-25T01:22:07.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd92191a640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c06c840 0x7fd91c15d890 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd910009fd0 tx=0x7fd91000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:07.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd90e7fc640 1 -- 172.21.15.77:0/1699417567 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910017070 con 0x7fd91c06c840 2024-04-25T01:22:07.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd90e7fc640 1 -- 172.21.15.77:0/1699417567 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd91000c940 con 0x7fd91c06c840 2024-04-25T01:22:07.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd923b7e640 1 -- 172.21.15.77:0/1699417567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd91c0cda20 con 0x7fd91c06c840 2024-04-25T01:22:07.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd90e7fc640 1 -- 172.21.15.77:0/1699417567 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91000cd10 con 0x7fd91c06c840 2024-04-25T01:22:07.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.349+0000 7fd923b7e640 1 -- 172.21.15.77:0/1699417567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd91c0cdfe0 con 0x7fd91c06c840 2024-04-25T01:22:07.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.350+0000 7fd923b7e640 1 -- 172.21.15.77:0/1699417567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd91c068fb0 con 0x7fd91c06c840 2024-04-25T01:22:07.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.351+0000 7fd90e7fc640 1 -- 172.21.15.77:0/1699417567 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd91001a070 con 0x7fd91c06c840 2024-04-25T01:22:07.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.351+0000 7fd90e7fc640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd90006be30 0x7fd90006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.351+0000 7fd90e7fc640 1 -- 172.21.15.77:0/1699417567 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd91008b040 con 0x7fd91c06c840 2024-04-25T01:22:07.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.351+0000 7fd921119640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd90006be30 0x7fd90006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.352+0000 7fd921119640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd90006be30 0x7fd90006e2f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd91c154510 tx=0x7fd918007e20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:07.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.355+0000 7fd90e7fc640 1 -- 172.21.15.77:0/1699417567 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd91c1f4f90 con 0x7fd91c06c840 2024-04-25T01:22:07.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.564+0000 7fd923b7e640 1 -- 172.21.15.77:0/1699417567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd91c157900 con 0x7fd91c06c840 2024-04-25T01:22:07.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.564+0000 7fd90e7fc640 1 -- 172.21.15.77:0/1699417567 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7fd910059060 con 0x7fd91c06c840 2024-04-25T01:22:07.546 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:07.547 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4267907621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4267907621},{"type":"v1","addr":"172.21.15.77:6837","nonce":4267907621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/493518847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":493518847},{"type":"v1","addr":"172.21.15.121:6833","nonce":493518847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:21:45.408154+0000","modified":"2024-04-25T01:22:06.554124+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6835/2810316639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2810316639},{"type":"v1","addr":"172.21.15.121:6835","nonce":2810316639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T01:22:07.547 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2024-04-25T01:22:07.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 -- 172.21.15.77:0/1699417567 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd90006be30 msgr2=0x7fd90006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd90006be30 0x7fd90006e2f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd91c154510 tx=0x7fd918007e20 comp rx=0 tx=0).stop 2024-04-25T01:22:07.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 -- 172.21.15.77:0/1699417567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c06c840 msgr2=0x7fd91c15d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c06c840 0x7fd91c15d890 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd910009fd0 tx=0x7fd91000ed40 comp rx=0 tx=0).stop 2024-04-25T01:22:07.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 -- 172.21.15.77:0/1699417567 shutdown_connections 2024-04-25T01:22:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd90006be30 0x7fd90006e2f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd91c0ade10 0x7fd91c0cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd91c06fcd0 0x7fd91c1f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 --2- 172.21.15.77:0/1699417567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c06c840 0x7fd91c15d890 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 -- 172.21.15.77:0/1699417567 >> 172.21.15.77:0/1699417567 conn(0x7fd91c071550 msgr2=0x7fd91c0b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 -- 172.21.15.77:0/1699417567 shutdown_connections 2024-04-25T01:22:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.566+0000 7fd923b7e640 1 -- 172.21.15.77:0/1699417567 wait complete. 2024-04-25T01:22:07.559 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:07.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.673+0000 7fa3056ee640 1 Processor -- start 2024-04-25T01:22:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.673+0000 7fa3056ee640 1 -- start start 2024-04-25T01:22:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.674+0000 7fa3056ee640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa3000cd2e0 0x7fa30006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.674+0000 7fa3056ee640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3000d0330 con 0x7fa30006fa50 2024-04-25T01:22:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.674+0000 7fa3056ee640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3000d0610 con 0x7fa3000ce500 2024-04-25T01:22:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.674+0000 7fa3056ee640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa3000d0f40 con 0x7fa3000cd2e0 2024-04-25T01:22:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.674+0000 7fa2fe7fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa30006fa50 0x7fa30015d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56380/0 (socket says 172.21.15.77:56380) 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.674+0000 7fa2fe7fc640 1 -- 172.21.15.77:0/4240709397 learned_addr learned my addr 172.21.15.77:0/4240709397 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.674+0000 7fa2ff7fe640 1 --2- 172.21.15.77:0/4240709397 >> v2:172.21.15.121:3300/0 conn(0x7fa3000cd2e0 0x7fa30006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.675+0000 7fa2fdffb640 1 -- 172.21.15.77:0/4240709397 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3668016706 0 0) 0x7fa3000d0330 con 0x7fa30006fa50 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.675+0000 7fa2fdffb640 1 -- 172.21.15.77:0/4240709397 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2e8003520 con 0x7fa30006fa50 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.675+0000 7fa2ff7fe640 1 -- 172.21.15.77:0/4240709397 >> v1:172.21.15.121:6789/0 conn(0x7fa3000ce500 legacy=0x7fa3000ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.675+0000 7fa2ff7fe640 1 -- 172.21.15.77:0/4240709397 >> v1:172.21.15.77:6789/0 conn(0x7fa30006fa50 legacy=0x7fa30015d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.675+0000 7fa2fdffb640 1 -- 172.21.15.77:0/4240709397 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2090482331 0 0) 0x7fa2e8003520 con 0x7fa30006fa50 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.675+0000 7fa2ff7fe640 1 -- 172.21.15.77:0/4240709397 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3000d08f0 con 0x7fa3000cd2e0 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.676+0000 7fa2ff7fe640 1 --2- 172.21.15.77:0/4240709397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3000cd2e0 0x7fa30006fee0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80097a0 tx=0x7fa2f802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e151465b3882a08f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.676+0000 7fa2fdffb640 1 -- 172.21.15.77:0/4240709397 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8037070 con 0x7fa3000cd2e0 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.676+0000 7fa2fdffb640 1 -- 172.21.15.77:0/4240709397 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2f802ac50 con 0x7fa3000cd2e0 2024-04-25T01:22:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.676+0000 7fa2fdffb640 1 -- 172.21.15.77:0/4240709397 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4001670 con 0x7fa3000cd2e0 2024-04-25T01:22:07.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.676+0000 7fa3056ee640 1 -- 172.21.15.77:0/4240709397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3000cd2e0 msgr2=0x7fa30006fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.676+0000 7fa3056ee640 1 --2- 172.21.15.77:0/4240709397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3000cd2e0 0x7fa30006fee0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80097a0 tx=0x7fa2f802a1c0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.676+0000 7fa3056ee640 1 -- 172.21.15.77:0/4240709397 shutdown_connections 2024-04-25T01:22:07.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.676+0000 7fa3056ee640 1 --2- 172.21.15.77:0/4240709397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3000cd2e0 0x7fa30006fee0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.676+0000 7fa3056ee640 1 -- 172.21.15.77:0/4240709397 >> 172.21.15.77:0/4240709397 conn(0x7fa3000d1c80 msgr2=0x7fa3000d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:07.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.676+0000 7fa3056ee640 1 -- 172.21.15.77:0/4240709397 shutdown_connections 2024-04-25T01:22:07.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.676+0000 7fa3056ee640 1 -- 172.21.15.77:0/4240709397 wait complete. 2024-04-25T01:22:07.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.678+0000 7fa3056ee640 1 Processor -- start 2024-04-25T01:22:07.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.678+0000 7fa3056ee640 1 -- start start 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.678+0000 7fa3056ee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa30006fa50 0x7fa300150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.678+0000 7fa3056ee640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3000cd2e0 0x7fa3000c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.678+0000 7fa3056ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3000ce500 0x7fa30015fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.678+0000 7fa3056ee640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa3000b1510 con 0x7fa3000ce500 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.678+0000 7fa3056ee640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa3000b1390 con 0x7fa3000cd2e0 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.678+0000 7fa3056ee640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa3000cfec0 con 0x7fa30006fa50 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.678+0000 7fa2ff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3000ce500 0x7fa30015fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.678+0000 7fa2ff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3000ce500 0x7fa30015fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56002/0 (socket says 172.21.15.77:56002) 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.678+0000 7fa2ff7fe640 1 -- 172.21.15.77:0/2075441625 learned_addr learned my addr 172.21.15.77:0/2075441625 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2fe7fc640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3000cd2e0 0x7fa3000c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2feffd640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa30006fa50 0x7fa300150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2ff7fe640 1 -- 172.21.15.77:0/2075441625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa30006fa50 msgr2=0x7fa300150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2ff7fe640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa30006fa50 0x7fa300150070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2ff7fe640 1 -- 172.21.15.77:0/2075441625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3000cd2e0 msgr2=0x7fa3000c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2ff7fe640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3000cd2e0 0x7fa3000c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2ff7fe640 1 -- 172.21.15.77:0/2075441625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3000b6eb0 con 0x7fa3000ce500 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2fe7fc640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3000cd2e0 0x7fa3000c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2ff7fe640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3000ce500 0x7fa30015fb00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8035040 tx=0x7fa2f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2e77fe640 1 -- 172.21.15.77:0/2075441625 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8041020 con 0x7fa3000ce500 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2e77fe640 1 -- 172.21.15.77:0/2075441625 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2f803d420 con 0x7fa3000ce500 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa3056ee640 1 -- 172.21.15.77:0/2075441625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3000b60f0 con 0x7fa3000ce500 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa2e77fe640 1 -- 172.21.15.77:0/2075441625 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8037040 con 0x7fa3000ce500 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.679+0000 7fa3056ee640 1 -- 172.21.15.77:0/2075441625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3000b66b0 con 0x7fa3000ce500 2024-04-25T01:22:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.680+0000 7fa3056ee640 1 -- 172.21.15.77:0/2075441625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa300068180 con 0x7fa3000ce500 2024-04-25T01:22:07.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.681+0000 7fa2e77fe640 1 -- 172.21.15.77:0/2075441625 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa2f8044020 con 0x7fa3000ce500 2024-04-25T01:22:07.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.681+0000 7fa2e77fe640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2ec06be30 0x7fa2ec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.681+0000 7fa2feffd640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2ec06be30 0x7fa2ec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.681+0000 7fa2e77fe640 1 -- 172.21.15.77:0/2075441625 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa2f80ab590 con 0x7fa3000ce500 2024-04-25T01:22:07.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.681+0000 7fa2feffd640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2ec06be30 0x7fa2ec06e2f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa2f00012e0 tx=0x7fa2f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:07.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.685+0000 7fa2e77fe640 1 -- 172.21.15.77:0/2075441625 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa2f804b030 con 0x7fa3000ce500 2024-04-25T01:22:07.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.860+0000 7fa3056ee640 1 -- 172.21.15.77:0/2075441625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3000b7c00 con 0x7fa3000ce500 2024-04-25T01:22:07.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.861+0000 7fa2e77fe640 1 -- 172.21.15.77:0/2075441625 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7fa2f8098070 con 0x7fa3000ce500 2024-04-25T01:22:07.843 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:07.844 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:07.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 -- 172.21.15.77:0/2075441625 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2ec06be30 msgr2=0x7fa2ec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2ec06be30 0x7fa2ec06e2f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa2f00012e0 tx=0x7fa2f0009040 comp rx=0 tx=0).stop 2024-04-25T01:22:07.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 -- 172.21.15.77:0/2075441625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3000ce500 msgr2=0x7fa30015fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3000ce500 0x7fa30015fb00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8035040 tx=0x7fa2f80046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 -- 172.21.15.77:0/2075441625 shutdown_connections 2024-04-25T01:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa2ec06be30 0x7fa2ec06e2f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3000ce500 0x7fa30015fb00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3000cd2e0 0x7fa3000c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 --2- 172.21.15.77:0/2075441625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa30006fa50 0x7fa300150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 -- 172.21.15.77:0/2075441625 >> 172.21.15.77:0/2075441625 conn(0x7fa3000d1c80 msgr2=0x7fa30006ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 -- 172.21.15.77:0/2075441625 shutdown_connections 2024-04-25T01:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.863+0000 7fa3056ee640 1 -- 172.21.15.77:0/2075441625 wait complete. 2024-04-25T01:22:07.856 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T01:22:07.857 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:22:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.971+0000 7fa735f42640 1 Processor -- start 2024-04-25T01:22:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.971+0000 7fa735f42640 1 -- start start 2024-04-25T01:22:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa735f42640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730154ac0 0x7fa73015b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa735f42640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa7300cf2a0 0x7fa7300711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa735f42640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa730154ea0 con 0x7fa730071730 2024-04-25T01:22:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa735f42640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa73006d790 con 0x7fa7300cf2a0 2024-04-25T01:22:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa735f42640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa73006da70 con 0x7fa730154ac0 2024-04-25T01:22:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa72effd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa7300cf2a0 0x7fa7300711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa72f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730154ac0 0x7fa73015b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa72ffff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa730071730 0x7fa73006ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:59576/0 (socket says 172.21.15.77:59576) 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa72f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730154ac0 0x7fa73015b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33576/0 (socket says 172.21.15.77:33576) 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa72ffff640 1 -- 172.21.15.77:0/914910011 learned_addr learned my addr 172.21.15.77:0/914910011 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa72f7fe640 1 -- 172.21.15.77:0/914910011 >> v2:172.21.15.121:3300/0 conn(0x7fa7300cf2a0 msgr2=0x7fa7300711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa72f7fe640 1 --2- 172.21.15.77:0/914910011 >> v2:172.21.15.121:3300/0 conn(0x7fa7300cf2a0 0x7fa7300711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.972+0000 7fa72f7fe640 1 -- 172.21.15.77:0/914910011 >> v1:172.21.15.121:6789/0 conn(0x7fa730071730 legacy=0x7fa73006ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.973+0000 7fa72e7fc640 1 -- 172.21.15.77:0/914910011 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1600583073 0 0) 0x7fa730154ea0 con 0x7fa730071730 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.973+0000 7fa72f7fe640 1 -- 172.21.15.77:0/914910011 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa73006dd50 con 0x7fa730154ac0 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.973+0000 7fa72effd640 1 --2- 172.21.15.77:0/914910011 >> v2:172.21.15.121:3300/0 conn(0x7fa7300cf2a0 0x7fa7300711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.973+0000 7fa72f7fe640 1 --2- 172.21.15.77:0/914910011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730154ac0 0x7fa73015b5f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa7200097a0 tx=0x7fa72002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ce0acf31174613a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.973+0000 7fa72e7fc640 1 -- 172.21.15.77:0/914910011 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa720037070 con 0x7fa730154ac0 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.973+0000 7fa72e7fc640 1 -- 172.21.15.77:0/914910011 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa720032440 con 0x7fa730154ac0 2024-04-25T01:22:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.973+0000 7fa72e7fc640 1 -- 172.21.15.77:0/914910011 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa720032810 con 0x7fa730154ac0 2024-04-25T01:22:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.974+0000 7fa735f42640 1 -- 172.21.15.77:0/914910011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730154ac0 msgr2=0x7fa73015b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.974+0000 7fa735f42640 1 --2- 172.21.15.77:0/914910011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730154ac0 0x7fa73015b5f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa7200097a0 tx=0x7fa72002a180 comp rx=0 tx=0).stop 2024-04-25T01:22:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.974+0000 7fa735f42640 1 -- 172.21.15.77:0/914910011 shutdown_connections 2024-04-25T01:22:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.974+0000 7fa735f42640 1 --2- 172.21.15.77:0/914910011 >> v2:172.21.15.121:3300/0 conn(0x7fa7300cf2a0 0x7fa7300711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.974+0000 7fa735f42640 1 --2- 172.21.15.77:0/914910011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730154ac0 0x7fa73015b5f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.974+0000 7fa735f42640 1 -- 172.21.15.77:0/914910011 >> 172.21.15.77:0/914910011 conn(0x7fa7300b1820 msgr2=0x7fa7300b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.974+0000 7fa735f42640 1 -- 172.21.15.77:0/914910011 shutdown_connections 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.974+0000 7fa735f42640 1 -- 172.21.15.77:0/914910011 wait complete. 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.975+0000 7fa735f42640 1 Processor -- start 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.975+0000 7fa735f42640 1 -- start start 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.976+0000 7fa735f42640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730071730 0x7fa7301636e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.976+0000 7fa735f42640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7300cf2a0 0x7fa730158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.976+0000 7fa735f42640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa730154ac0 0x7fa730159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.976+0000 7fa735f42640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa73006d3c0 con 0x7fa730154ac0 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.976+0000 7fa735f42640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa7300cf720 con 0x7fa7300cf2a0 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.976+0000 7fa735f42640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa73006d540 con 0x7fa730071730 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.976+0000 7fa72ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa730154ac0 0x7fa730159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.976+0000 7fa72ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa730154ac0 0x7fa730159500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56010/0 (socket says 172.21.15.77:56010) 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.976+0000 7fa72ffff640 1 -- 172.21.15.77:0/3952805499 learned_addr learned my addr 172.21.15.77:0/3952805499 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.976+0000 7fa72effd640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7300cf2a0 0x7fa730158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.976+0000 7fa72f7fe640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730071730 0x7fa7301636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa72ffff640 1 -- 172.21.15.77:0/3952805499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730071730 msgr2=0x7fa7301636e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa72ffff640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730071730 0x7fa7301636e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa72ffff640 1 -- 172.21.15.77:0/3952805499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7300cf2a0 msgr2=0x7fa730158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa72ffff640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7300cf2a0 0x7fa730158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa72ffff640 1 -- 172.21.15.77:0/3952805499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa730159c80 con 0x7fa730154ac0 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa72f7fe640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730071730 0x7fa7301636e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa72effd640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7300cf2a0 0x7fa730158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa72ffff640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa730154ac0 0x7fa730159500 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa72800c7a0 tx=0x7fa72800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa717fff640 1 -- 172.21.15.77:0/3952805499 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa728018070 con 0x7fa730154ac0 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa717fff640 1 -- 172.21.15.77:0/3952805499 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa72800a640 con 0x7fa730154ac0 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa735f42640 1 -- 172.21.15.77:0/3952805499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa730159f60 con 0x7fa730154ac0 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa717fff640 1 -- 172.21.15.77:0/3952805499 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72800aab0 con 0x7fa730154ac0 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.977+0000 7fa735f42640 1 -- 172.21.15.77:0/3952805499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa73015a520 con 0x7fa730154ac0 2024-04-25T01:22:07.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.978+0000 7fa735f42640 1 -- 172.21.15.77:0/3952805499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa730068fb0 con 0x7fa730154ac0 2024-04-25T01:22:07.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.978+0000 7fa717fff640 1 -- 172.21.15.77:0/3952805499 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa72801b070 con 0x7fa730154ac0 2024-04-25T01:22:07.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.979+0000 7fa717fff640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa71c06be60 0x7fa71c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.979+0000 7fa717fff640 1 -- 172.21.15.77:0/3952805499 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa72808a720 con 0x7fa730154ac0 2024-04-25T01:22:07.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.979+0000 7fa72f7fe640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa71c06be60 0x7fa71c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.979+0000 7fa72f7fe640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa71c06be60 0x7fa71c06e320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa720034000 tx=0x7fa72002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:07.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:07.983+0000 7fa717fff640 1 -- 172.21.15.77:0/3952805499 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa728058e20 con 0x7fa730154ac0 2024-04-25T01:22:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.154+0000 7fa735f42640 1 -- 172.21.15.77:0/3952805499 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa73015b3d0 con 0x7fa71c06be60 2024-04-25T01:22:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.162+0000 7fa717fff640 1 -- 172.21.15.77:0/3952805499 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7fa73015b3d0 con 0x7fa71c06be60 2024-04-25T01:22:08.148 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 -- 172.21.15.77:0/3952805499 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa71c06be60 msgr2=0x7fa71c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa71c06be60 0x7fa71c06e320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa720034000 tx=0x7fa72002e000 comp rx=0 tx=0).stop 2024-04-25T01:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 -- 172.21.15.77:0/3952805499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa730154ac0 msgr2=0x7fa730159500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa730154ac0 0x7fa730159500 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa72800c7a0 tx=0x7fa72800cc60 comp rx=0 tx=0).stop 2024-04-25T01:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 -- 172.21.15.77:0/3952805499 shutdown_connections 2024-04-25T01:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa71c06be60 0x7fa71c06e320 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa730154ac0 0x7fa730159500 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7300cf2a0 0x7fa730158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 --2- 172.21.15.77:0/3952805499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa730071730 0x7fa7301636e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 -- 172.21.15.77:0/3952805499 >> 172.21.15.77:0/3952805499 conn(0x7fa7300b1820 msgr2=0x7fa73006ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 -- 172.21.15.77:0/3952805499 shutdown_connections 2024-04-25T01:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.166+0000 7fa735f42640 1 -- 172.21.15.77:0/3952805499 wait complete. 2024-04-25T01:22:08.158 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-25T01:22:07.160553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219400,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755320,"statfs":{"total":773094113280,"available":772869447680,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003046"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471974+0000","last_change":"2024-04-25T01:21:45.082729+0000","last_active":"2024-04-25T01:21:51.471974+0000","last_peered":"2024-04-25T01:21:51.471974+0000","last_clean":"2024-04-25T01:21:51.471974+0000","last_became_active":"2024-04-25T01:21:45.082441+0000","last_became_peered":"2024-04-25T01:21:45.082441+0000","last_unstale":"2024-04-25T01:21:51.471974+0000","last_undegraded":"2024-04-25T01:21:51.471974+0000","last_fullsized":"2024-04-25T01:21:51.471974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:31.228997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472236+0000","last_change":"2024-04-25T01:21:46.093885+0000","last_active":"2024-04-25T01:21:51.472236+0000","last_peered":"2024-04-25T01:21:51.472236+0000","last_clean":"2024-04-25T01:21:51.472236+0000","last_became_active":"2024-04-25T01:21:46.092142+0000","last_became_peered":"2024-04-25T01:21:46.092142+0000","last_unstale":"2024-04-25T01:21:51.472236+0000","last_undegraded":"2024-04-25T01:21:51.472236+0000","last_fullsized":"2024-04-25T01:21:51.472236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471738+0000","last_change":"2024-04-25T01:21:50.187217+0000","last_active":"2024-04-25T01:21:51.471738+0000","last_peered":"2024-04-25T01:21:51.471738+0000","last_clean":"2024-04-25T01:21:51.471738+0000","last_became_active":"2024-04-25T01:21:50.186698+0000","last_became_peered":"2024-04-25T01:21:50.186698+0000","last_unstale":"2024-04-25T01:21:51.471738+0000","last_undegraded":"2024-04-25T01:21:51.471738+0000","last_fullsized":"2024-04-25T01:21:51.471738+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:50.627648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472373+0000","last_change":"2024-04-25T01:21:45.078191+0000","last_active":"2024-04-25T01:21:51.472373+0000","last_peered":"2024-04-25T01:21:51.472373+0000","last_clean":"2024-04-25T01:21:51.472373+0000","last_became_active":"2024-04-25T01:21:45.077949+0000","last_became_peered":"2024-04-25T01:21:45.077949+0000","last_unstale":"2024-04-25T01:21:51.472373+0000","last_undegraded":"2024-04-25T01:21:51.472373+0000","last_fullsized":"2024-04-25T01:21:51.472373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:23.371789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:46.091354+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:46.090827+0000","last_became_peered":"2024-04-25T01:21:46.090827+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.117971+0000","last_change":"2024-04-25T01:21:50.126246+0000","last_active":"2024-04-25T01:21:51.117971+0000","last_peered":"2024-04-25T01:21:51.117971+0000","last_clean":"2024-04-25T01:21:51.117971+0000","last_became_active":"2024-04-25T01:21:50.125823+0000","last_became_peered":"2024-04-25T01:21:50.125823+0000","last_unstale":"2024-04-25T01:21:51.117971+0000","last_undegraded":"2024-04-25T01:21:51.117971+0000","last_fullsized":"2024-04-25T01:21:51.117971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:05.469540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472877+0000","last_change":"2024-04-25T01:21:45.080811+0000","last_active":"2024-04-25T01:21:51.472877+0000","last_peered":"2024-04-25T01:21:51.472877+0000","last_clean":"2024-04-25T01:21:51.472877+0000","last_became_active":"2024-04-25T01:21:45.079902+0000","last_became_peered":"2024-04-25T01:21:45.079902+0000","last_unstale":"2024-04-25T01:21:51.472877+0000","last_undegraded":"2024-04-25T01:21:51.472877+0000","last_fullsized":"2024-04-25T01:21:51.472877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:15.208981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472902+0000","last_change":"2024-04-25T01:21:46.090986+0000","last_active":"2024-04-25T01:21:51.472902+0000","last_peered":"2024-04-25T01:21:51.472902+0000","last_clean":"2024-04-25T01:21:51.472902+0000","last_became_active":"2024-04-25T01:21:46.090715+0000","last_became_peered":"2024-04-25T01:21:46.090715+0000","last_unstale":"2024-04-25T01:21:51.472902+0000","last_undegraded":"2024-04-25T01:21:51.472902+0000","last_fullsized":"2024-04-25T01:21:51.472902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:27.583834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118940+0000","last_change":"2024-04-25T01:21:50.128507+0000","last_active":"2024-04-25T01:21:51.118940+0000","last_peered":"2024-04-25T01:21:51.118940+0000","last_clean":"2024-04-25T01:21:51.118940+0000","last_became_active":"2024-04-25T01:21:50.128116+0000","last_became_peered":"2024-04-25T01:21:50.128116+0000","last_unstale":"2024-04-25T01:21:51.118940+0000","last_undegraded":"2024-04-25T01:21:51.118940+0000","last_fullsized":"2024-04-25T01:21:51.118940+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.077404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472303+0000","last_change":"2024-04-25T01:21:45.080401+0000","last_active":"2024-04-25T01:21:51.472303+0000","last_peered":"2024-04-25T01:21:51.472303+0000","last_clean":"2024-04-25T01:21:51.472303+0000","last_became_active":"2024-04-25T01:21:45.079961+0000","last_became_peered":"2024-04-25T01:21:45.079961+0000","last_unstale":"2024-04-25T01:21:51.472303+0000","last_undegraded":"2024-04-25T01:21:51.472303+0000","last_fullsized":"2024-04-25T01:21:51.472303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473268+0000","last_change":"2024-04-25T01:21:46.090524+0000","last_active":"2024-04-25T01:21:51.473268+0000","last_peered":"2024-04-25T01:21:51.473268+0000","last_clean":"2024-04-25T01:21:51.473268+0000","last_became_active":"2024-04-25T01:21:46.089869+0000","last_became_peered":"2024-04-25T01:21:46.089869+0000","last_unstale":"2024-04-25T01:21:51.473268+0000","last_undegraded":"2024-04-25T01:21:51.473268+0000","last_fullsized":"2024-04-25T01:21:51.473268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:24.344315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118039+0000","last_change":"2024-04-25T01:21:50.118075+0000","last_active":"2024-04-25T01:21:51.118039+0000","last_peered":"2024-04-25T01:21:51.118039+0000","last_clean":"2024-04-25T01:21:51.118039+0000","last_became_active":"2024-04-25T01:21:50.117657+0000","last_became_peered":"2024-04-25T01:21:50.117657+0000","last_unstale":"2024-04-25T01:21:51.118039+0000","last_undegraded":"2024-04-25T01:21:51.118039+0000","last_fullsized":"2024-04-25T01:21:51.118039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:29.510529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118231+0000","last_change":"2024-04-25T01:21:46.093011+0000","last_active":"2024-04-25T01:21:51.118231+0000","last_peered":"2024-04-25T01:21:51.118231+0000","last_clean":"2024-04-25T01:21:51.118231+0000","last_became_active":"2024-04-25T01:21:46.092710+0000","last_became_peered":"2024-04-25T01:21:46.092710+0000","last_unstale":"2024-04-25T01:21:51.118231+0000","last_undegraded":"2024-04-25T01:21:51.118231+0000","last_fullsized":"2024-04-25T01:21:51.118231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472603+0000","last_change":"2024-04-25T01:21:45.080010+0000","last_active":"2024-04-25T01:21:51.472603+0000","last_peered":"2024-04-25T01:21:51.472603+0000","last_clean":"2024-04-25T01:21:51.472603+0000","last_became_active":"2024-04-25T01:21:45.079732+0000","last_became_peered":"2024-04-25T01:21:45.079732+0000","last_unstale":"2024-04-25T01:21:51.472603+0000","last_undegraded":"2024-04-25T01:21:51.472603+0000","last_fullsized":"2024-04-25T01:21:51.472603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119330+0000","last_change":"2024-04-25T01:21:50.121929+0000","last_active":"2024-04-25T01:21:51.119330+0000","last_peered":"2024-04-25T01:21:51.119330+0000","last_clean":"2024-04-25T01:21:51.119330+0000","last_became_active":"2024-04-25T01:21:50.121288+0000","last_became_peered":"2024-04-25T01:21:50.121288+0000","last_unstale":"2024-04-25T01:21:51.119330+0000","last_undegraded":"2024-04-25T01:21:51.119330+0000","last_fullsized":"2024-04-25T01:21:51.119330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473407+0000","last_change":"2024-04-25T01:21:46.082691+0000","last_active":"2024-04-25T01:21:51.473407+0000","last_peered":"2024-04-25T01:21:51.473407+0000","last_clean":"2024-04-25T01:21:51.473407+0000","last_became_active":"2024-04-25T01:21:46.082266+0000","last_became_peered":"2024-04-25T01:21:46.082266+0000","last_unstale":"2024-04-25T01:21:51.473407+0000","last_undegraded":"2024-04-25T01:21:51.473407+0000","last_fullsized":"2024-04-25T01:21:51.473407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:22.971333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473925+0000","last_change":"2024-04-25T01:21:45.076031+0000","last_active":"2024-04-25T01:21:51.473925+0000","last_peered":"2024-04-25T01:21:51.473925+0000","last_clean":"2024-04-25T01:21:51.473925+0000","last_became_active":"2024-04-25T01:21:45.075769+0000","last_became_peered":"2024-04-25T01:21:45.075769+0000","last_unstale":"2024-04-25T01:21:51.473925+0000","last_undegraded":"2024-04-25T01:21:51.473925+0000","last_fullsized":"2024-04-25T01:21:51.473925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118699+0000","last_change":"2024-04-25T01:21:50.122266+0000","last_active":"2024-04-25T01:21:51.118699+0000","last_peered":"2024-04-25T01:21:51.118699+0000","last_clean":"2024-04-25T01:21:51.118699+0000","last_became_active":"2024-04-25T01:21:50.121879+0000","last_became_peered":"2024-04-25T01:21:50.121879+0000","last_unstale":"2024-04-25T01:21:51.118699+0000","last_undegraded":"2024-04-25T01:21:51.118699+0000","last_fullsized":"2024-04-25T01:21:51.118699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:51.825101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471956+0000","last_change":"2024-04-25T01:21:46.093175+0000","last_active":"2024-04-25T01:21:51.471956+0000","last_peered":"2024-04-25T01:21:51.471956+0000","last_clean":"2024-04-25T01:21:51.471956+0000","last_became_active":"2024-04-25T01:21:46.092327+0000","last_became_peered":"2024-04-25T01:21:46.092327+0000","last_unstale":"2024-04-25T01:21:51.471956+0000","last_undegraded":"2024-04-25T01:21:51.471956+0000","last_fullsized":"2024-04-25T01:21:51.471956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473162+0000","last_change":"2024-04-25T01:21:45.077148+0000","last_active":"2024-04-25T01:21:51.473162+0000","last_peered":"2024-04-25T01:21:51.473162+0000","last_clean":"2024-04-25T01:21:51.473162+0000","last_became_active":"2024-04-25T01:21:45.076858+0000","last_became_peered":"2024-04-25T01:21:45.076858+0000","last_unstale":"2024-04-25T01:21:51.473162+0000","last_undegraded":"2024-04-25T01:21:51.473162+0000","last_fullsized":"2024-04-25T01:21:51.473162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:15.208981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473795+0000","last_change":"2024-04-25T01:21:50.182249+0000","last_active":"2024-04-25T01:21:51.473795+0000","last_peered":"2024-04-25T01:21:51.473795+0000","last_clean":"2024-04-25T01:21:51.473795+0000","last_became_active":"2024-04-25T01:21:50.181860+0000","last_became_peered":"2024-04-25T01:21:50.181860+0000","last_unstale":"2024-04-25T01:21:51.473795+0000","last_undegraded":"2024-04-25T01:21:51.473795+0000","last_fullsized":"2024-04-25T01:21:51.473795+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:38.806941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119159+0000","last_change":"2024-04-25T01:21:46.085766+0000","last_active":"2024-04-25T01:21:51.119159+0000","last_peered":"2024-04-25T01:21:51.119159+0000","last_clean":"2024-04-25T01:21:51.119159+0000","last_became_active":"2024-04-25T01:21:46.085479+0000","last_became_peered":"2024-04-25T01:21:46.085479+0000","last_unstale":"2024-04-25T01:21:51.119159+0000","last_undegraded":"2024-04-25T01:21:51.119159+0000","last_fullsized":"2024-04-25T01:21:51.119159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:02.493076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:45.085085+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:45.084796+0000","last_became_peered":"2024-04-25T01:21:45.084796+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:25.950316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118843+0000","last_change":"2024-04-25T01:21:50.123379+0000","last_active":"2024-04-25T01:21:51.118843+0000","last_peered":"2024-04-25T01:21:51.118843+0000","last_clean":"2024-04-25T01:21:51.118843+0000","last_became_active":"2024-04-25T01:21:50.123057+0000","last_became_peered":"2024-04-25T01:21:50.123057+0000","last_unstale":"2024-04-25T01:21:51.118843+0000","last_undegraded":"2024-04-25T01:21:51.118843+0000","last_fullsized":"2024-04-25T01:21:51.118843+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:13.828811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472820+0000","last_change":"2024-04-25T01:21:46.090075+0000","last_active":"2024-04-25T01:21:51.472820+0000","last_peered":"2024-04-25T01:21:51.472820+0000","last_clean":"2024-04-25T01:21:51.472820+0000","last_became_active":"2024-04-25T01:21:46.089800+0000","last_became_peered":"2024-04-25T01:21:46.089800+0000","last_unstale":"2024-04-25T01:21:51.472820+0000","last_undegraded":"2024-04-25T01:21:51.472820+0000","last_fullsized":"2024-04-25T01:21:51.472820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119254+0000","last_change":"2024-04-25T01:21:45.080102+0000","last_active":"2024-04-25T01:21:51.119254+0000","last_peered":"2024-04-25T01:21:51.119254+0000","last_clean":"2024-04-25T01:21:51.119254+0000","last_became_active":"2024-04-25T01:21:45.079786+0000","last_became_peered":"2024-04-25T01:21:45.079786+0000","last_unstale":"2024-04-25T01:21:51.119254+0000","last_undegraded":"2024-04-25T01:21:51.119254+0000","last_fullsized":"2024-04-25T01:21:51.119254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:51.235948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118619+0000","last_change":"2024-04-25T01:21:50.185727+0000","last_active":"2024-04-25T01:21:51.118619+0000","last_peered":"2024-04-25T01:21:51.118619+0000","last_clean":"2024-04-25T01:21:51.118619+0000","last_became_active":"2024-04-25T01:21:50.184395+0000","last_became_peered":"2024-04-25T01:21:50.184395+0000","last_unstale":"2024-04-25T01:21:51.118619+0000","last_undegraded":"2024-04-25T01:21:51.118619+0000","last_fullsized":"2024-04-25T01:21:51.118619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:05.257957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472627+0000","last_change":"2024-04-25T01:21:46.099799+0000","last_active":"2024-04-25T01:21:51.472627+0000","last_peered":"2024-04-25T01:21:51.472627+0000","last_clean":"2024-04-25T01:21:51.472627+0000","last_became_active":"2024-04-25T01:21:46.099506+0000","last_became_peered":"2024-04-25T01:21:46.099506+0000","last_unstale":"2024-04-25T01:21:51.472627+0000","last_undegraded":"2024-04-25T01:21:51.472627+0000","last_fullsized":"2024-04-25T01:21:51.472627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473591+0000","last_change":"2024-04-25T01:21:45.079966+0000","last_active":"2024-04-25T01:21:51.473591+0000","last_peered":"2024-04-25T01:21:51.473591+0000","last_clean":"2024-04-25T01:21:51.473591+0000","last_became_active":"2024-04-25T01:21:45.079689+0000","last_became_peered":"2024-04-25T01:21:45.079689+0000","last_unstale":"2024-04-25T01:21:51.473591+0000","last_undegraded":"2024-04-25T01:21:51.473591+0000","last_fullsized":"2024-04-25T01:21:51.473591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471654+0000","last_change":"2024-04-25T01:21:50.184865+0000","last_active":"2024-04-25T01:21:51.471654+0000","last_peered":"2024-04-25T01:21:51.471654+0000","last_clean":"2024-04-25T01:21:51.471654+0000","last_became_active":"2024-04-25T01:21:50.184466+0000","last_became_peered":"2024-04-25T01:21:50.184466+0000","last_unstale":"2024-04-25T01:21:51.471654+0000","last_undegraded":"2024-04-25T01:21:51.471654+0000","last_fullsized":"2024-04-25T01:21:51.471654+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:42.606728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473876+0000","last_change":"2024-04-25T01:21:46.100915+0000","last_active":"2024-04-25T01:21:51.473876+0000","last_peered":"2024-04-25T01:21:51.473876+0000","last_clean":"2024-04-25T01:21:51.473876+0000","last_became_active":"2024-04-25T01:21:46.100641+0000","last_became_peered":"2024-04-25T01:21:46.100641+0000","last_unstale":"2024-04-25T01:21:51.473876+0000","last_undegraded":"2024-04-25T01:21:51.473876+0000","last_fullsized":"2024-04-25T01:21:51.473876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472835+0000","last_change":"2024-04-25T01:21:45.085103+0000","last_active":"2024-04-25T01:21:51.472835+0000","last_peered":"2024-04-25T01:21:51.472835+0000","last_clean":"2024-04-25T01:21:51.472835+0000","last_became_active":"2024-04-25T01:21:45.084805+0000","last_became_peered":"2024-04-25T01:21:45.084805+0000","last_unstale":"2024-04-25T01:21:51.472835+0000","last_undegraded":"2024-04-25T01:21:51.472835+0000","last_fullsized":"2024-04-25T01:21:51.472835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:56.444547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471942+0000","last_change":"2024-04-25T01:21:50.184815+0000","last_active":"2024-04-25T01:21:51.471942+0000","last_peered":"2024-04-25T01:21:51.471942+0000","last_clean":"2024-04-25T01:21:51.471942+0000","last_became_active":"2024-04-25T01:21:50.184441+0000","last_became_peered":"2024-04-25T01:21:50.184441+0000","last_unstale":"2024-04-25T01:21:51.471942+0000","last_undegraded":"2024-04-25T01:21:51.471942+0000","last_fullsized":"2024-04-25T01:21:51.471942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473364+0000","last_change":"2024-04-25T01:21:46.096428+0000","last_active":"2024-04-25T01:21:51.473364+0000","last_peered":"2024-04-25T01:21:51.473364+0000","last_clean":"2024-04-25T01:21:51.473364+0000","last_became_active":"2024-04-25T01:21:46.096151+0000","last_became_peered":"2024-04-25T01:21:46.096151+0000","last_unstale":"2024-04-25T01:21:51.473364+0000","last_undegraded":"2024-04-25T01:21:51.473364+0000","last_fullsized":"2024-04-25T01:21:51.473364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:42.961329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473960+0000","last_change":"2024-04-25T01:21:45.080814+0000","last_active":"2024-04-25T01:21:51.473960+0000","last_peered":"2024-04-25T01:21:51.473960+0000","last_clean":"2024-04-25T01:21:51.473960+0000","last_became_active":"2024-04-25T01:21:45.080505+0000","last_became_peered":"2024-04-25T01:21:45.080505+0000","last_unstale":"2024-04-25T01:21:51.473960+0000","last_undegraded":"2024-04-25T01:21:51.473960+0000","last_fullsized":"2024-04-25T01:21:51.473960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473549+0000","last_change":"2024-04-25T01:21:50.129319+0000","last_active":"2024-04-25T01:21:51.473549+0000","last_peered":"2024-04-25T01:21:51.473549+0000","last_clean":"2024-04-25T01:21:51.473549+0000","last_became_active":"2024-04-25T01:21:50.128797+0000","last_became_peered":"2024-04-25T01:21:50.128797+0000","last_unstale":"2024-04-25T01:21:51.473549+0000","last_undegraded":"2024-04-25T01:21:51.473549+0000","last_fullsized":"2024-04-25T01:21:51.473549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:31.623179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119209+0000","last_change":"2024-04-25T01:21:46.079764+0000","last_active":"2024-04-25T01:21:51.119209+0000","last_peered":"2024-04-25T01:21:51.119209+0000","last_clean":"2024-04-25T01:21:51.119209+0000","last_became_active":"2024-04-25T01:21:46.079475+0000","last_became_peered":"2024-04-25T01:21:46.079475+0000","last_unstale":"2024-04-25T01:21:51.119209+0000","last_undegraded":"2024-04-25T01:21:51.119209+0000","last_fullsized":"2024-04-25T01:21:51.119209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:21.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473306+0000","last_change":"2024-04-25T01:21:45.075554+0000","last_active":"2024-04-25T01:21:51.473306+0000","last_peered":"2024-04-25T01:21:51.473306+0000","last_clean":"2024-04-25T01:21:51.473306+0000","last_became_active":"2024-04-25T01:21:45.075256+0000","last_became_peered":"2024-04-25T01:21:45.075256+0000","last_unstale":"2024-04-25T01:21:51.473306+0000","last_undegraded":"2024-04-25T01:21:51.473306+0000","last_fullsized":"2024-04-25T01:21:51.473306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473829+0000","last_change":"2024-04-25T01:21:50.131544+0000","last_active":"2024-04-25T01:21:51.473829+0000","last_peered":"2024-04-25T01:21:51.473829+0000","last_clean":"2024-04-25T01:21:51.473829+0000","last_became_active":"2024-04-25T01:21:50.131125+0000","last_became_peered":"2024-04-25T01:21:50.131125+0000","last_unstale":"2024-04-25T01:21:51.473829+0000","last_undegraded":"2024-04-25T01:21:51.473829+0000","last_fullsized":"2024-04-25T01:21:51.473829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:00.042952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472863+0000","last_change":"2024-04-25T01:21:46.094965+0000","last_active":"2024-04-25T01:21:51.472863+0000","last_peered":"2024-04-25T01:21:51.472863+0000","last_clean":"2024-04-25T01:21:51.472863+0000","last_became_active":"2024-04-25T01:21:46.094515+0000","last_became_peered":"2024-04-25T01:21:46.094515+0000","last_unstale":"2024-04-25T01:21:51.472863+0000","last_undegraded":"2024-04-25T01:21:51.472863+0000","last_fullsized":"2024-04-25T01:21:51.472863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119246+0000","last_change":"2024-04-25T01:21:45.075901+0000","last_active":"2024-04-25T01:21:51.119246+0000","last_peered":"2024-04-25T01:21:51.119246+0000","last_clean":"2024-04-25T01:21:51.119246+0000","last_became_active":"2024-04-25T01:21:45.075635+0000","last_became_peered":"2024-04-25T01:21:45.075635+0000","last_unstale":"2024-04-25T01:21:51.119246+0000","last_undegraded":"2024-04-25T01:21:51.119246+0000","last_fullsized":"2024-04-25T01:21:51.119246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473778+0000","last_change":"2024-04-25T01:21:50.129249+0000","last_active":"2024-04-25T01:21:51.473778+0000","last_peered":"2024-04-25T01:21:51.473778+0000","last_clean":"2024-04-25T01:21:51.473778+0000","last_became_active":"2024-04-25T01:21:50.128867+0000","last_became_peered":"2024-04-25T01:21:50.128867+0000","last_unstale":"2024-04-25T01:21:51.473778+0000","last_undegraded":"2024-04-25T01:21:51.473778+0000","last_fullsized":"2024-04-25T01:21:51.473778+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:50.627648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471590+0000","last_change":"2024-04-25T01:21:46.078894+0000","last_active":"2024-04-25T01:21:51.471590+0000","last_peered":"2024-04-25T01:21:51.471590+0000","last_clean":"2024-04-25T01:21:51.471590+0000","last_became_active":"2024-04-25T01:21:46.078598+0000","last_became_peered":"2024-04-25T01:21:46.078598+0000","last_unstale":"2024-04-25T01:21:51.471590+0000","last_undegraded":"2024-04-25T01:21:51.471590+0000","last_fullsized":"2024-04-25T01:21:51.471590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:57.449234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471807+0000","last_change":"2024-04-25T01:21:45.080835+0000","last_active":"2024-04-25T01:21:51.471807+0000","last_peered":"2024-04-25T01:21:51.471807+0000","last_clean":"2024-04-25T01:21:51.471807+0000","last_became_active":"2024-04-25T01:21:45.080544+0000","last_became_peered":"2024-04-25T01:21:45.080544+0000","last_unstale":"2024-04-25T01:21:51.471807+0000","last_undegraded":"2024-04-25T01:21:51.471807+0000","last_fullsized":"2024-04-25T01:21:51.471807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472752+0000","last_change":"2024-04-25T01:21:50.182264+0000","last_active":"2024-04-25T01:21:51.472752+0000","last_peered":"2024-04-25T01:21:51.472752+0000","last_clean":"2024-04-25T01:21:51.472752+0000","last_became_active":"2024-04-25T01:21:50.181854+0000","last_became_peered":"2024-04-25T01:21:50.181854+0000","last_unstale":"2024-04-25T01:21:51.472752+0000","last_undegraded":"2024-04-25T01:21:51.472752+0000","last_fullsized":"2024-04-25T01:21:51.472752+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:28.383661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:46.102152+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:46.101917+0000","last_became_peered":"2024-04-25T01:21:46.101917+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473111+0000","last_change":"2024-04-25T01:21:45.070971+0000","last_active":"2024-04-25T01:21:51.473111+0000","last_peered":"2024-04-25T01:21:51.473111+0000","last_clean":"2024-04-25T01:21:51.473111+0000","last_became_active":"2024-04-25T01:21:45.070626+0000","last_became_peered":"2024-04-25T01:21:45.070626+0000","last_unstale":"2024-04-25T01:21:51.473111+0000","last_undegraded":"2024-04-25T01:21:51.473111+0000","last_fullsized":"2024-04-25T01:21:51.473111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471866+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:21:51.471866+0000","last_peered":"2024-04-25T01:21:51.471866+0000","last_clean":"2024-04-25T01:21:51.471866+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:21:51.471866+0000","last_undegraded":"2024-04-25T01:21:51.471866+0000","last_fullsized":"2024-04-25T01:21:51.471866+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118918+0000","last_change":"2024-04-25T01:21:50.185591+0000","last_active":"2024-04-25T01:21:51.118918+0000","last_peered":"2024-04-25T01:21:51.118918+0000","last_clean":"2024-04-25T01:21:51.118918+0000","last_became_active":"2024-04-25T01:21:50.185206+0000","last_became_peered":"2024-04-25T01:21:50.185206+0000","last_unstale":"2024-04-25T01:21:51.118918+0000","last_undegraded":"2024-04-25T01:21:51.118918+0000","last_fullsized":"2024-04-25T01:21:51.118918+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.077404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472462+0000","last_change":"2024-04-25T01:21:46.104388+0000","last_active":"2024-04-25T01:21:51.472462+0000","last_peered":"2024-04-25T01:21:51.472462+0000","last_clean":"2024-04-25T01:21:51.472462+0000","last_became_active":"2024-04-25T01:21:46.103864+0000","last_became_peered":"2024-04-25T01:21:46.103864+0000","last_unstale":"2024-04-25T01:21:51.472462+0000","last_undegraded":"2024-04-25T01:21:51.472462+0000","last_fullsized":"2024-04-25T01:21:51.472462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:46.038059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119490+0000","last_change":"2024-04-25T01:21:45.078985+0000","last_active":"2024-04-25T01:21:51.119490+0000","last_peered":"2024-04-25T01:21:51.119490+0000","last_clean":"2024-04-25T01:21:51.119490+0000","last_became_active":"2024-04-25T01:21:45.078782+0000","last_became_peered":"2024-04-25T01:21:45.078782+0000","last_unstale":"2024-04-25T01:21:51.119490+0000","last_undegraded":"2024-04-25T01:21:51.119490+0000","last_fullsized":"2024-04-25T01:21:51.119490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:51.235948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118175+0000","last_change":"2024-04-25T01:21:50.182278+0000","last_active":"2024-04-25T01:21:51.118175+0000","last_peered":"2024-04-25T01:21:51.118175+0000","last_clean":"2024-04-25T01:21:51.118175+0000","last_became_active":"2024-04-25T01:21:50.181838+0000","last_became_peered":"2024-04-25T01:21:50.181838+0000","last_unstale":"2024-04-25T01:21:51.118175+0000","last_undegraded":"2024-04-25T01:21:51.118175+0000","last_fullsized":"2024-04-25T01:21:51.118175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:13.828811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473013+0000","last_change":"2024-04-25T01:21:46.096433+0000","last_active":"2024-04-25T01:21:51.473013+0000","last_peered":"2024-04-25T01:21:51.473013+0000","last_clean":"2024-04-25T01:21:51.473013+0000","last_became_active":"2024-04-25T01:21:46.096197+0000","last_became_peered":"2024-04-25T01:21:46.096197+0000","last_unstale":"2024-04-25T01:21:51.473013+0000","last_undegraded":"2024-04-25T01:21:51.473013+0000","last_fullsized":"2024-04-25T01:21:51.473013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:38.567383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473546+0000","last_change":"2024-04-25T01:21:45.072096+0000","last_active":"2024-04-25T01:21:51.473546+0000","last_peered":"2024-04-25T01:21:51.473546+0000","last_clean":"2024-04-25T01:21:51.473546+0000","last_became_active":"2024-04-25T01:21:45.071811+0000","last_became_peered":"2024-04-25T01:21:45.071811+0000","last_unstale":"2024-04-25T01:21:51.473546+0000","last_undegraded":"2024-04-25T01:21:51.473546+0000","last_fullsized":"2024-04-25T01:21:51.473546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472183+0000","last_change":"2024-04-25T01:21:50.123158+0000","last_active":"2024-04-25T01:21:51.472183+0000","last_peered":"2024-04-25T01:21:51.472183+0000","last_clean":"2024-04-25T01:21:51.472183+0000","last_became_active":"2024-04-25T01:21:50.122275+0000","last_became_peered":"2024-04-25T01:21:50.122275+0000","last_unstale":"2024-04-25T01:21:51.472183+0000","last_undegraded":"2024-04-25T01:21:51.472183+0000","last_fullsized":"2024-04-25T01:21:51.472183+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:49.064038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472847+0000","last_change":"2024-04-25T01:21:46.105159+0000","last_active":"2024-04-25T01:21:51.472847+0000","last_peered":"2024-04-25T01:21:51.472847+0000","last_clean":"2024-04-25T01:21:51.472847+0000","last_became_active":"2024-04-25T01:21:46.104876+0000","last_became_peered":"2024-04-25T01:21:46.104876+0000","last_unstale":"2024-04-25T01:21:51.472847+0000","last_undegraded":"2024-04-25T01:21:51.472847+0000","last_fullsized":"2024-04-25T01:21:51.472847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:38.567383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119570+0000","last_change":"2024-04-25T01:21:45.076578+0000","last_active":"2024-04-25T01:21:51.119570+0000","last_peered":"2024-04-25T01:21:51.119570+0000","last_clean":"2024-04-25T01:21:51.119570+0000","last_became_active":"2024-04-25T01:21:45.076022+0000","last_became_peered":"2024-04-25T01:21:45.076022+0000","last_unstale":"2024-04-25T01:21:51.119570+0000","last_undegraded":"2024-04-25T01:21:51.119570+0000","last_fullsized":"2024-04-25T01:21:51.119570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471952+0000","last_change":"2024-04-25T01:21:50.182219+0000","last_active":"2024-04-25T01:21:51.471952+0000","last_peered":"2024-04-25T01:21:51.471952+0000","last_clean":"2024-04-25T01:21:51.471952+0000","last_became_active":"2024-04-25T01:21:50.181820+0000","last_became_peered":"2024-04-25T01:21:50.181820+0000","last_unstale":"2024-04-25T01:21:51.471952+0000","last_undegraded":"2024-04-25T01:21:51.471952+0000","last_fullsized":"2024-04-25T01:21:51.471952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:20.747339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472586+0000","last_change":"2024-04-25T01:21:46.100017+0000","last_active":"2024-04-25T01:21:51.472586+0000","last_peered":"2024-04-25T01:21:51.472586+0000","last_clean":"2024-04-25T01:21:51.472586+0000","last_became_active":"2024-04-25T01:21:46.099745+0000","last_became_peered":"2024-04-25T01:21:46.099745+0000","last_unstale":"2024-04-25T01:21:51.472586+0000","last_undegraded":"2024-04-25T01:21:51.472586+0000","last_fullsized":"2024-04-25T01:21:51.472586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:30.138160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118932+0000","last_change":"2024-04-25T01:21:45.081659+0000","last_active":"2024-04-25T01:21:51.118932+0000","last_peered":"2024-04-25T01:21:51.118932+0000","last_clean":"2024-04-25T01:21:51.118932+0000","last_became_active":"2024-04-25T01:21:45.081320+0000","last_became_peered":"2024-04-25T01:21:45.081320+0000","last_unstale":"2024-04-25T01:21:51.118932+0000","last_undegraded":"2024-04-25T01:21:51.118932+0000","last_fullsized":"2024-04-25T01:21:51.118932+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:11:25.817031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118175+0000","last_change":"2024-04-25T01:21:50.122840+0000","last_active":"2024-04-25T01:21:51.118175+0000","last_peered":"2024-04-25T01:21:51.118175+0000","last_clean":"2024-04-25T01:21:51.118175+0000","last_became_active":"2024-04-25T01:21:50.122184+0000","last_became_peered":"2024-04-25T01:21:50.122184+0000","last_unstale":"2024-04-25T01:21:51.118175+0000","last_undegraded":"2024-04-25T01:21:51.118175+0000","last_fullsized":"2024-04-25T01:21:51.118175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:27.010679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472223+0000","last_change":"2024-04-25T01:21:46.105988+0000","last_active":"2024-04-25T01:21:51.472223+0000","last_peered":"2024-04-25T01:21:51.472223+0000","last_clean":"2024-04-25T01:21:51.472223+0000","last_became_active":"2024-04-25T01:21:46.105479+0000","last_became_peered":"2024-04-25T01:21:46.105479+0000","last_unstale":"2024-04-25T01:21:51.472223+0000","last_undegraded":"2024-04-25T01:21:51.472223+0000","last_fullsized":"2024-04-25T01:21:51.472223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:45.024693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119108+0000","last_change":"2024-04-25T01:21:45.074194+0000","last_active":"2024-04-25T01:21:51.119108+0000","last_peered":"2024-04-25T01:21:51.119108+0000","last_clean":"2024-04-25T01:21:51.119108+0000","last_became_active":"2024-04-25T01:21:45.073923+0000","last_became_peered":"2024-04-25T01:21:45.073923+0000","last_unstale":"2024-04-25T01:21:51.119108+0000","last_undegraded":"2024-04-25T01:21:51.119108+0000","last_fullsized":"2024-04-25T01:21:51.119108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472699+0000","last_change":"2024-04-25T01:21:50.126685+0000","last_active":"2024-04-25T01:21:51.472699+0000","last_peered":"2024-04-25T01:21:51.472699+0000","last_clean":"2024-04-25T01:21:51.472699+0000","last_became_active":"2024-04-25T01:21:50.126183+0000","last_became_peered":"2024-04-25T01:21:50.126183+0000","last_unstale":"2024-04-25T01:21:51.472699+0000","last_undegraded":"2024-04-25T01:21:51.472699+0000","last_fullsized":"2024-04-25T01:21:51.472699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.177506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118469+0000","last_change":"2024-04-25T01:21:46.096531+0000","last_active":"2024-04-25T01:21:51.118469+0000","last_peered":"2024-04-25T01:21:51.118469+0000","last_clean":"2024-04-25T01:21:51.118469+0000","last_became_active":"2024-04-25T01:21:46.096250+0000","last_became_peered":"2024-04-25T01:21:46.096250+0000","last_unstale":"2024-04-25T01:21:51.118469+0000","last_undegraded":"2024-04-25T01:21:51.118469+0000","last_fullsized":"2024-04-25T01:21:51.118469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:55.076328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.117985+0000","last_change":"2024-04-25T01:21:45.079574+0000","last_active":"2024-04-25T01:21:51.117985+0000","last_peered":"2024-04-25T01:21:51.117985+0000","last_clean":"2024-04-25T01:21:51.117985+0000","last_became_active":"2024-04-25T01:21:45.079306+0000","last_became_peered":"2024-04-25T01:21:45.079306+0000","last_unstale":"2024-04-25T01:21:51.117985+0000","last_undegraded":"2024-04-25T01:21:51.117985+0000","last_fullsized":"2024-04-25T01:21:51.117985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472488+0000","last_change":"2024-04-25T01:21:50.187252+0000","last_active":"2024-04-25T01:21:51.472488+0000","last_peered":"2024-04-25T01:21:51.472488+0000","last_clean":"2024-04-25T01:21:51.472488+0000","last_became_active":"2024-04-25T01:21:50.186509+0000","last_became_peered":"2024-04-25T01:21:50.186509+0000","last_unstale":"2024-04-25T01:21:51.472488+0000","last_undegraded":"2024-04-25T01:21:51.472488+0000","last_fullsized":"2024-04-25T01:21:51.472488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:15.374624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472403+0000","last_change":"2024-04-25T01:21:46.104318+0000","last_active":"2024-04-25T01:21:51.472403+0000","last_peered":"2024-04-25T01:21:51.472403+0000","last_clean":"2024-04-25T01:21:51.472403+0000","last_became_active":"2024-04-25T01:21:46.103937+0000","last_became_peered":"2024-04-25T01:21:46.103937+0000","last_unstale":"2024-04-25T01:21:51.472403+0000","last_undegraded":"2024-04-25T01:21:51.472403+0000","last_fullsized":"2024-04-25T01:21:51.472403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:27.802903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473263+0000","last_change":"2024-04-25T01:21:45.076486+0000","last_active":"2024-04-25T01:21:51.473263+0000","last_peered":"2024-04-25T01:21:51.473263+0000","last_clean":"2024-04-25T01:21:51.473263+0000","last_became_active":"2024-04-25T01:21:45.076164+0000","last_became_peered":"2024-04-25T01:21:45.076164+0000","last_unstale":"2024-04-25T01:21:51.473263+0000","last_undegraded":"2024-04-25T01:21:51.473263+0000","last_fullsized":"2024-04-25T01:21:51.473263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472905+0000","last_change":"2024-04-25T01:21:50.182316+0000","last_active":"2024-04-25T01:21:51.472905+0000","last_peered":"2024-04-25T01:21:51.472905+0000","last_clean":"2024-04-25T01:21:51.472905+0000","last_became_active":"2024-04-25T01:21:50.181918+0000","last_became_peered":"2024-04-25T01:21:50.181918+0000","last_unstale":"2024-04-25T01:21:51.472905+0000","last_undegraded":"2024-04-25T01:21:51.472905+0000","last_fullsized":"2024-04-25T01:21:51.472905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:37.041783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472461+0000","last_change":"2024-04-25T01:21:46.100833+0000","last_active":"2024-04-25T01:21:51.472461+0000","last_peered":"2024-04-25T01:21:51.472461+0000","last_clean":"2024-04-25T01:21:51.472461+0000","last_became_active":"2024-04-25T01:21:46.100591+0000","last_became_peered":"2024-04-25T01:21:46.100591+0000","last_unstale":"2024-04-25T01:21:51.472461+0000","last_undegraded":"2024-04-25T01:21:51.472461+0000","last_fullsized":"2024-04-25T01:21:51.472461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:36.537071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472415+0000","last_change":"2024-04-25T01:21:45.078469+0000","last_active":"2024-04-25T01:21:51.472415+0000","last_peered":"2024-04-25T01:21:51.472415+0000","last_clean":"2024-04-25T01:21:51.472415+0000","last_became_active":"2024-04-25T01:21:45.078231+0000","last_became_peered":"2024-04-25T01:21:45.078231+0000","last_unstale":"2024-04-25T01:21:51.472415+0000","last_undegraded":"2024-04-25T01:21:51.472415+0000","last_fullsized":"2024-04-25T01:21:51.472415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:23.371789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472786+0000","last_change":"2024-04-25T01:21:50.185207+0000","last_active":"2024-04-25T01:21:51.472786+0000","last_peered":"2024-04-25T01:21:51.472786+0000","last_clean":"2024-04-25T01:21:51.472786+0000","last_became_active":"2024-04-25T01:21:50.184793+0000","last_became_peered":"2024-04-25T01:21:50.184793+0000","last_unstale":"2024-04-25T01:21:51.472786+0000","last_undegraded":"2024-04-25T01:21:51.472786+0000","last_fullsized":"2024-04-25T01:21:51.472786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:15.374624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472438+0000","last_change":"2024-04-25T01:21:46.104460+0000","last_active":"2024-04-25T01:21:51.472438+0000","last_peered":"2024-04-25T01:21:51.472438+0000","last_clean":"2024-04-25T01:21:51.472438+0000","last_became_active":"2024-04-25T01:21:46.104163+0000","last_became_peered":"2024-04-25T01:21:46.104163+0000","last_unstale":"2024-04-25T01:21:51.472438+0000","last_undegraded":"2024-04-25T01:21:51.472438+0000","last_fullsized":"2024-04-25T01:21:51.472438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:27.583834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:55.386967+0000","last_change":"2024-04-25T01:21:45.076242+0000","last_active":"2024-04-25T01:21:55.386967+0000","last_peered":"2024-04-25T01:21:55.386967+0000","last_clean":"2024-04-25T01:21:55.386967+0000","last_became_active":"2024-04-25T01:21:45.075952+0000","last_became_peered":"2024-04-25T01:21:45.075952+0000","last_unstale":"2024-04-25T01:21:55.386967+0000","last_undegraded":"2024-04-25T01:21:55.386967+0000","last_fullsized":"2024-04-25T01:21:55.386967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:52.481389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473757+0000","last_change":"2024-04-25T01:21:50.128837+0000","last_active":"2024-04-25T01:21:51.473757+0000","last_peered":"2024-04-25T01:21:51.473757+0000","last_clean":"2024-04-25T01:21:51.473757+0000","last_became_active":"2024-04-25T01:21:50.128462+0000","last_became_peered":"2024-04-25T01:21:50.128462+0000","last_unstale":"2024-04-25T01:21:51.473757+0000","last_undegraded":"2024-04-25T01:21:51.473757+0000","last_fullsized":"2024-04-25T01:21:51.473757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:42.606728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118967+0000","last_change":"2024-04-25T01:21:46.100341+0000","last_active":"2024-04-25T01:21:51.118967+0000","last_peered":"2024-04-25T01:21:51.118967+0000","last_clean":"2024-04-25T01:21:51.118967+0000","last_became_active":"2024-04-25T01:21:46.099961+0000","last_became_peered":"2024-04-25T01:21:46.099961+0000","last_unstale":"2024-04-25T01:21:51.118967+0000","last_undegraded":"2024-04-25T01:21:51.118967+0000","last_fullsized":"2024-04-25T01:21:51.118967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:15.176543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473450+0000","last_change":"2024-04-25T01:21:45.077748+0000","last_active":"2024-04-25T01:21:51.473450+0000","last_peered":"2024-04-25T01:21:51.473450+0000","last_clean":"2024-04-25T01:21:51.473450+0000","last_became_active":"2024-04-25T01:21:45.077494+0000","last_became_peered":"2024-04-25T01:21:45.077494+0000","last_unstale":"2024-04-25T01:21:51.473450+0000","last_undegraded":"2024-04-25T01:21:51.473450+0000","last_fullsized":"2024-04-25T01:21:51.473450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:25.950316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118695+0000","last_change":"2024-04-25T01:21:50.126572+0000","last_active":"2024-04-25T01:21:51.118695+0000","last_peered":"2024-04-25T01:21:51.118695+0000","last_clean":"2024-04-25T01:21:51.118695+0000","last_became_active":"2024-04-25T01:21:50.126013+0000","last_became_peered":"2024-04-25T01:21:50.126013+0000","last_unstale":"2024-04-25T01:21:51.118695+0000","last_undegraded":"2024-04-25T01:21:51.118695+0000","last_fullsized":"2024-04-25T01:21:51.118695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:19.877429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471878+0000","last_change":"2024-04-25T01:21:46.091135+0000","last_active":"2024-04-25T01:21:51.471878+0000","last_peered":"2024-04-25T01:21:51.471878+0000","last_clean":"2024-04-25T01:21:51.471878+0000","last_became_active":"2024-04-25T01:21:46.090493+0000","last_became_peered":"2024-04-25T01:21:46.090493+0000","last_unstale":"2024-04-25T01:21:51.471878+0000","last_undegraded":"2024-04-25T01:21:51.471878+0000","last_fullsized":"2024-04-25T01:21:51.471878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472529+0000","last_change":"2024-04-25T01:21:45.075179+0000","last_active":"2024-04-25T01:21:51.472529+0000","last_peered":"2024-04-25T01:21:51.472529+0000","last_clean":"2024-04-25T01:21:51.472529+0000","last_became_active":"2024-04-25T01:21:45.074903+0000","last_became_peered":"2024-04-25T01:21:45.074903+0000","last_unstale":"2024-04-25T01:21:51.472529+0000","last_undegraded":"2024-04-25T01:21:51.472529+0000","last_fullsized":"2024-04-25T01:21:51.472529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471819+0000","last_change":"2024-04-25T01:21:50.126406+0000","last_active":"2024-04-25T01:21:51.471819+0000","last_peered":"2024-04-25T01:21:51.471819+0000","last_clean":"2024-04-25T01:21:51.471819+0000","last_became_active":"2024-04-25T01:21:50.126029+0000","last_became_peered":"2024-04-25T01:21:50.126029+0000","last_unstale":"2024-04-25T01:21:51.471819+0000","last_undegraded":"2024-04-25T01:21:51.471819+0000","last_fullsized":"2024-04-25T01:21:51.471819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:38.806941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472398+0000","last_change":"2024-04-25T01:21:46.100746+0000","last_active":"2024-04-25T01:21:51.472398+0000","last_peered":"2024-04-25T01:21:51.472398+0000","last_clean":"2024-04-25T01:21:51.472398+0000","last_became_active":"2024-04-25T01:21:46.100475+0000","last_became_peered":"2024-04-25T01:21:46.100475+0000","last_unstale":"2024-04-25T01:21:51.472398+0000","last_undegraded":"2024-04-25T01:21:51.472398+0000","last_fullsized":"2024-04-25T01:21:51.472398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:27.802903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119162+0000","last_change":"2024-04-25T01:21:45.082455+0000","last_active":"2024-04-25T01:21:51.119162+0000","last_peered":"2024-04-25T01:21:51.119162+0000","last_clean":"2024-04-25T01:21:51.119162+0000","last_became_active":"2024-04-25T01:21:45.082135+0000","last_became_peered":"2024-04-25T01:21:45.082135+0000","last_unstale":"2024-04-25T01:21:51.119162+0000","last_undegraded":"2024-04-25T01:21:51.119162+0000","last_fullsized":"2024-04-25T01:21:51.119162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:06.296153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471896+0000","last_change":"2024-04-25T01:21:50.184859+0000","last_active":"2024-04-25T01:21:51.471896+0000","last_peered":"2024-04-25T01:21:51.471896+0000","last_clean":"2024-04-25T01:21:51.471896+0000","last_became_active":"2024-04-25T01:21:50.184475+0000","last_became_peered":"2024-04-25T01:21:50.184475+0000","last_unstale":"2024-04-25T01:21:51.471896+0000","last_undegraded":"2024-04-25T01:21:51.471896+0000","last_fullsized":"2024-04-25T01:21:51.471896+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:00.042952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472589+0000","last_change":"2024-04-25T01:21:50.184894+0000","last_active":"2024-04-25T01:21:51.472589+0000","last_peered":"2024-04-25T01:21:51.472589+0000","last_clean":"2024-04-25T01:21:51.472589+0000","last_became_active":"2024-04-25T01:21:50.184392+0000","last_became_peered":"2024-04-25T01:21:50.184392+0000","last_unstale":"2024-04-25T01:21:51.472589+0000","last_undegraded":"2024-04-25T01:21:51.472589+0000","last_fullsized":"2024-04-25T01:21:51.472589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:31.842249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472948+0000","last_change":"2024-04-25T01:21:46.092888+0000","last_active":"2024-04-25T01:21:51.472948+0000","last_peered":"2024-04-25T01:21:51.472948+0000","last_clean":"2024-04-25T01:21:51.472948+0000","last_became_active":"2024-04-25T01:21:46.092607+0000","last_became_peered":"2024-04-25T01:21:46.092607+0000","last_unstale":"2024-04-25T01:21:51.472948+0000","last_undegraded":"2024-04-25T01:21:51.472948+0000","last_fullsized":"2024-04-25T01:21:51.472948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:22.971333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472329+0000","last_change":"2024-04-25T01:21:45.070532+0000","last_active":"2024-04-25T01:21:51.472329+0000","last_peered":"2024-04-25T01:21:51.472329+0000","last_clean":"2024-04-25T01:21:51.472329+0000","last_became_active":"2024-04-25T01:21:45.070256+0000","last_became_peered":"2024-04-25T01:21:45.070256+0000","last_unstale":"2024-04-25T01:21:51.472329+0000","last_undegraded":"2024-04-25T01:21:51.472329+0000","last_fullsized":"2024-04-25T01:21:51.472329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119151+0000","last_change":"2024-04-25T01:21:50.130056+0000","last_active":"2024-04-25T01:21:51.119151+0000","last_peered":"2024-04-25T01:21:51.119151+0000","last_clean":"2024-04-25T01:21:51.119151+0000","last_became_active":"2024-04-25T01:21:50.129541+0000","last_became_peered":"2024-04-25T01:21:50.129541+0000","last_unstale":"2024-04-25T01:21:51.119151+0000","last_undegraded":"2024-04-25T01:21:51.119151+0000","last_fullsized":"2024-04-25T01:21:51.119151+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471871+0000","last_change":"2024-04-25T01:21:46.083728+0000","last_active":"2024-04-25T01:21:51.471871+0000","last_peered":"2024-04-25T01:21:51.471871+0000","last_clean":"2024-04-25T01:21:51.471871+0000","last_became_active":"2024-04-25T01:21:46.083458+0000","last_became_peered":"2024-04-25T01:21:46.083458+0000","last_unstale":"2024-04-25T01:21:51.471871+0000","last_undegraded":"2024-04-25T01:21:51.471871+0000","last_fullsized":"2024-04-25T01:21:51.471871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472093+0000","last_change":"2024-04-25T01:21:45.077308+0000","last_active":"2024-04-25T01:21:51.472093+0000","last_peered":"2024-04-25T01:21:51.472093+0000","last_clean":"2024-04-25T01:21:51.472093+0000","last_became_active":"2024-04-25T01:21:45.076815+0000","last_became_peered":"2024-04-25T01:21:45.076815+0000","last_unstale":"2024-04-25T01:21:51.472093+0000","last_undegraded":"2024-04-25T01:21:51.472093+0000","last_fullsized":"2024-04-25T01:21:51.472093+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:43.833096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118432+0000","last_change":"2024-04-25T01:21:45.079268+0000","last_active":"2024-04-25T01:21:51.118432+0000","last_peered":"2024-04-25T01:21:51.118432+0000","last_clean":"2024-04-25T01:21:51.118432+0000","last_became_active":"2024-04-25T01:21:45.078934+0000","last_became_peered":"2024-04-25T01:21:45.078934+0000","last_unstale":"2024-04-25T01:21:51.118432+0000","last_undegraded":"2024-04-25T01:21:51.118432+0000","last_fullsized":"2024-04-25T01:21:51.118432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472316+0000","last_change":"2024-04-25T01:21:46.086329+0000","last_active":"2024-04-25T01:21:51.472316+0000","last_peered":"2024-04-25T01:21:51.472316+0000","last_clean":"2024-04-25T01:21:51.472316+0000","last_became_active":"2024-04-25T01:21:46.086069+0000","last_became_peered":"2024-04-25T01:21:46.086069+0000","last_unstale":"2024-04-25T01:21:51.472316+0000","last_undegraded":"2024-04-25T01:21:51.472316+0000","last_fullsized":"2024-04-25T01:21:51.472316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119441+0000","last_change":"2024-04-25T01:21:50.122334+0000","last_active":"2024-04-25T01:21:51.119441+0000","last_peered":"2024-04-25T01:21:51.119441+0000","last_clean":"2024-04-25T01:21:51.119441+0000","last_became_active":"2024-04-25T01:21:50.121974+0000","last_became_peered":"2024-04-25T01:21:50.121974+0000","last_unstale":"2024-04-25T01:21:51.119441+0000","last_undegraded":"2024-04-25T01:21:51.119441+0000","last_fullsized":"2024-04-25T01:21:51.119441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:59.115674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473488+0000","last_change":"2024-04-25T01:21:45.082740+0000","last_active":"2024-04-25T01:21:51.473488+0000","last_peered":"2024-04-25T01:21:51.473488+0000","last_clean":"2024-04-25T01:21:51.473488+0000","last_became_active":"2024-04-25T01:21:45.082439+0000","last_became_peered":"2024-04-25T01:21:45.082439+0000","last_unstale":"2024-04-25T01:21:51.473488+0000","last_undegraded":"2024-04-25T01:21:51.473488+0000","last_fullsized":"2024-04-25T01:21:51.473488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:52.481389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118237+0000","last_change":"2024-04-25T01:21:46.088483+0000","last_active":"2024-04-25T01:21:51.118237+0000","last_peered":"2024-04-25T01:21:51.118237+0000","last_clean":"2024-04-25T01:21:51.118237+0000","last_became_active":"2024-04-25T01:21:46.088113+0000","last_became_peered":"2024-04-25T01:21:46.088113+0000","last_unstale":"2024-04-25T01:21:51.118237+0000","last_undegraded":"2024-04-25T01:21:51.118237+0000","last_fullsized":"2024-04-25T01:21:51.118237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:22.055684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472316+0000","last_change":"2024-04-25T01:21:50.125319+0000","last_active":"2024-04-25T01:21:51.472316+0000","last_peered":"2024-04-25T01:21:51.472316+0000","last_clean":"2024-04-25T01:21:51.472316+0000","last_became_active":"2024-04-25T01:21:50.124926+0000","last_became_peered":"2024-04-25T01:21:50.124926+0000","last_unstale":"2024-04-25T01:21:51.472316+0000","last_undegraded":"2024-04-25T01:21:51.472316+0000","last_fullsized":"2024-04-25T01:21:51.472316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:20.747339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118296+0000","last_change":"2024-04-25T01:21:46.093690+0000","last_active":"2024-04-25T01:21:51.118296+0000","last_peered":"2024-04-25T01:21:51.118296+0000","last_clean":"2024-04-25T01:21:51.118296+0000","last_became_active":"2024-04-25T01:21:46.093392+0000","last_became_peered":"2024-04-25T01:21:46.093392+0000","last_unstale":"2024-04-25T01:21:51.118296+0000","last_undegraded":"2024-04-25T01:21:51.118296+0000","last_fullsized":"2024-04-25T01:21:51.118296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119144+0000","last_change":"2024-04-25T01:21:46.089771+0000","last_active":"2024-04-25T01:21:51.119144+0000","last_peered":"2024-04-25T01:21:51.119144+0000","last_clean":"2024-04-25T01:21:51.119144+0000","last_became_active":"2024-04-25T01:21:46.089489+0000","last_became_peered":"2024-04-25T01:21:46.089489+0000","last_unstale":"2024-04-25T01:21:51.119144+0000","last_undegraded":"2024-04-25T01:21:51.119144+0000","last_fullsized":"2024-04-25T01:21:51.119144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:16.213667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473158+0000","last_change":"2024-04-25T01:21:46.099812+0000","last_active":"2024-04-25T01:21:51.473158+0000","last_peered":"2024-04-25T01:21:51.473158+0000","last_clean":"2024-04-25T01:21:51.473158+0000","last_became_active":"2024-04-25T01:21:46.099508+0000","last_became_peered":"2024-04-25T01:21:46.099508+0000","last_unstale":"2024-04-25T01:21:51.473158+0000","last_undegraded":"2024-04-25T01:21:51.473158+0000","last_fullsized":"2024-04-25T01:21:51.473158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:45.024693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472233+0000","last_change":"2024-04-25T01:21:46.083637+0000","last_active":"2024-04-25T01:21:51.472233+0000","last_peered":"2024-04-25T01:21:51.472233+0000","last_clean":"2024-04-25T01:21:51.472233+0000","last_became_active":"2024-04-25T01:21:46.083371+0000","last_became_peered":"2024-04-25T01:21:46.083371+0000","last_unstale":"2024-04-25T01:21:51.472233+0000","last_undegraded":"2024-04-25T01:21:51.472233+0000","last_fullsized":"2024-04-25T01:21:51.472233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473043+0000","last_change":"2024-04-25T01:21:46.087822+0000","last_active":"2024-04-25T01:21:51.473043+0000","last_peered":"2024-04-25T01:21:51.473043+0000","last_clean":"2024-04-25T01:21:51.473043+0000","last_became_active":"2024-04-25T01:21:46.087555+0000","last_became_peered":"2024-04-25T01:21:46.087555+0000","last_unstale":"2024-04-25T01:21:51.473043+0000","last_undegraded":"2024-04-25T01:21:51.473043+0000","last_fullsized":"2024-04-25T01:21:51.473043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473261+0000","last_change":"2024-04-25T01:21:46.092978+0000","last_active":"2024-04-25T01:21:51.473261+0000","last_peered":"2024-04-25T01:21:51.473261+0000","last_clean":"2024-04-25T01:21:51.473261+0000","last_became_active":"2024-04-25T01:21:46.092720+0000","last_became_peered":"2024-04-25T01:21:46.092720+0000","last_unstale":"2024-04-25T01:21:51.473261+0000","last_undegraded":"2024-04-25T01:21:51.473261+0000","last_fullsized":"2024-04-25T01:21:51.473261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119314+0000","last_change":"2024-04-25T01:21:46.100969+0000","last_active":"2024-04-25T01:21:51.119314+0000","last_peered":"2024-04-25T01:21:51.119314+0000","last_clean":"2024-04-25T01:21:51.119314+0000","last_became_active":"2024-04-25T01:21:46.100687+0000","last_became_peered":"2024-04-25T01:21:46.100687+0000","last_unstale":"2024-04-25T01:21:51.119314+0000","last_undegraded":"2024-04-25T01:21:51.119314+0000","last_fullsized":"2024-04-25T01:21:51.119314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472539+0000","last_change":"2024-04-25T01:21:46.087688+0000","last_active":"2024-04-25T01:21:51.472539+0000","last_peered":"2024-04-25T01:21:51.472539+0000","last_clean":"2024-04-25T01:21:51.472539+0000","last_became_active":"2024-04-25T01:21:46.087422+0000","last_became_peered":"2024-04-25T01:21:46.087422+0000","last_unstale":"2024-04-25T01:21:51.472539+0000","last_undegraded":"2024-04-25T01:21:51.472539+0000","last_fullsized":"2024-04-25T01:21:51.472539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472217+0000","last_change":"2024-04-25T01:21:46.096585+0000","last_active":"2024-04-25T01:21:51.472217+0000","last_peered":"2024-04-25T01:21:51.472217+0000","last_clean":"2024-04-25T01:21:51.472217+0000","last_became_active":"2024-04-25T01:21:46.096299+0000","last_became_peered":"2024-04-25T01:21:46.096299+0000","last_unstale":"2024-04-25T01:21:51.472217+0000","last_undegraded":"2024-04-25T01:21:51.472217+0000","last_fullsized":"2024-04-25T01:21:51.472217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472814+0000","last_change":"2024-04-25T01:21:46.085048+0000","last_active":"2024-04-25T01:21:51.472814+0000","last_peered":"2024-04-25T01:21:51.472814+0000","last_clean":"2024-04-25T01:21:51.472814+0000","last_became_active":"2024-04-25T01:21:46.084766+0000","last_became_peered":"2024-04-25T01:21:46.084766+0000","last_unstale":"2024-04-25T01:21:51.472814+0000","last_undegraded":"2024-04-25T01:21:51.472814+0000","last_fullsized":"2024-04-25T01:21:51.472814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473111+0000","last_change":"2024-04-25T01:21:46.105035+0000","last_active":"2024-04-25T01:21:51.473111+0000","last_peered":"2024-04-25T01:21:51.473111+0000","last_clean":"2024-04-25T01:21:51.473111+0000","last_became_active":"2024-04-25T01:21:46.104776+0000","last_became_peered":"2024-04-25T01:21:46.104776+0000","last_unstale":"2024-04-25T01:21:51.473111+0000","last_undegraded":"2024-04-25T01:21:51.473111+0000","last_fullsized":"2024-04-25T01:21:51.473111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471992+0000","last_change":"2024-04-25T01:21:46.100414+0000","last_active":"2024-04-25T01:21:51.471992+0000","last_peered":"2024-04-25T01:21:51.471992+0000","last_clean":"2024-04-25T01:21:51.471992+0000","last_became_active":"2024-04-25T01:21:46.100140+0000","last_became_peered":"2024-04-25T01:21:46.100140+0000","last_unstale":"2024-04-25T01:21:51.471992+0000","last_undegraded":"2024-04-25T01:21:51.471992+0000","last_fullsized":"2024-04-25T01:21:51.471992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:55.076328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118784+0000","last_change":"2024-04-25T01:21:46.092761+0000","last_active":"2024-04-25T01:21:51.118784+0000","last_peered":"2024-04-25T01:21:51.118784+0000","last_clean":"2024-04-25T01:21:51.118784+0000","last_became_active":"2024-04-25T01:21:46.092230+0000","last_became_peered":"2024-04-25T01:21:46.092230+0000","last_unstale":"2024-04-25T01:21:51.118784+0000","last_undegraded":"2024-04-25T01:21:51.118784+0000","last_fullsized":"2024-04-25T01:21:51.118784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472724+0000","last_change":"2024-04-25T01:21:46.104951+0000","last_active":"2024-04-25T01:21:51.472724+0000","last_peered":"2024-04-25T01:21:51.472724+0000","last_clean":"2024-04-25T01:21:51.472724+0000","last_became_active":"2024-04-25T01:21:46.104662+0000","last_became_peered":"2024-04-25T01:21:46.104662+0000","last_unstale":"2024-04-25T01:21:51.472724+0000","last_undegraded":"2024-04-25T01:21:51.472724+0000","last_fullsized":"2024-04-25T01:21:51.472724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:46.588303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119527+0000","last_change":"2024-04-25T01:21:46.091860+0000","last_active":"2024-04-25T01:21:51.119527+0000","last_peered":"2024-04-25T01:21:51.119527+0000","last_clean":"2024-04-25T01:21:51.119527+0000","last_became_active":"2024-04-25T01:21:46.091569+0000","last_became_peered":"2024-04-25T01:21:46.091569+0000","last_unstale":"2024-04-25T01:21:51.119527+0000","last_undegraded":"2024-04-25T01:21:51.119527+0000","last_fullsized":"2024-04-25T01:21:51.119527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:57.449234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473437+0000","last_change":"2024-04-25T01:21:46.099956+0000","last_active":"2024-04-25T01:21:51.473437+0000","last_peered":"2024-04-25T01:21:51.473437+0000","last_clean":"2024-04-25T01:21:51.473437+0000","last_became_active":"2024-04-25T01:21:46.099687+0000","last_became_peered":"2024-04-25T01:21:46.099687+0000","last_unstale":"2024-04-25T01:21:51.473437+0000","last_undegraded":"2024-04-25T01:21:51.473437+0000","last_fullsized":"2024-04-25T01:21:51.473437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:30.138160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472108+0000","last_change":"2024-04-25T01:21:46.102084+0000","last_active":"2024-04-25T01:21:51.472108+0000","last_peered":"2024-04-25T01:21:51.472108+0000","last_clean":"2024-04-25T01:21:51.472108+0000","last_became_active":"2024-04-25T01:21:46.101539+0000","last_became_peered":"2024-04-25T01:21:46.101539+0000","last_unstale":"2024-04-25T01:21:51.472108+0000","last_undegraded":"2024-04-25T01:21:51.472108+0000","last_fullsized":"2024-04-25T01:21:51.472108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473063+0000","last_change":"2024-04-25T01:21:46.089149+0000","last_active":"2024-04-25T01:21:51.473063+0000","last_peered":"2024-04-25T01:21:51.473063+0000","last_clean":"2024-04-25T01:21:51.473063+0000","last_became_active":"2024-04-25T01:21:46.088845+0000","last_became_peered":"2024-04-25T01:21:46.088845+0000","last_unstale":"2024-04-25T01:21:51.473063+0000","last_undegraded":"2024-04-25T01:21:51.473063+0000","last_fullsized":"2024-04-25T01:21:51.473063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472391+0000","last_change":"2024-04-25T01:21:46.087764+0000","last_active":"2024-04-25T01:21:51.472391+0000","last_peered":"2024-04-25T01:21:51.472391+0000","last_clean":"2024-04-25T01:21:51.472391+0000","last_became_active":"2024-04-25T01:21:46.087481+0000","last_became_peered":"2024-04-25T01:21:46.087481+0000","last_unstale":"2024-04-25T01:21:51.472391+0000","last_undegraded":"2024-04-25T01:21:51.472391+0000","last_fullsized":"2024-04-25T01:21:51.472391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:16.213667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473322+0000","last_change":"2024-04-25T01:21:46.093981+0000","last_active":"2024-04-25T01:21:51.473322+0000","last_peered":"2024-04-25T01:21:51.473322+0000","last_clean":"2024-04-25T01:21:51.473322+0000","last_became_active":"2024-04-25T01:21:46.093729+0000","last_became_peered":"2024-04-25T01:21:46.093729+0000","last_unstale":"2024-04-25T01:21:51.473322+0000","last_undegraded":"2024-04-25T01:21:51.473322+0000","last_fullsized":"2024-04-25T01:21:51.473322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118866+0000","last_change":"2024-04-25T01:21:46.094258+0000","last_active":"2024-04-25T01:21:51.118866+0000","last_peered":"2024-04-25T01:21:51.118866+0000","last_clean":"2024-04-25T01:21:51.118866+0000","last_became_active":"2024-04-25T01:21:46.093633+0000","last_became_peered":"2024-04-25T01:21:46.093633+0000","last_unstale":"2024-04-25T01:21:51.118866+0000","last_undegraded":"2024-04-25T01:21:51.118866+0000","last_fullsized":"2024-04-25T01:21:51.118866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:52.240635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118920+0000","last_change":"2024-04-25T01:21:46.093764+0000","last_active":"2024-04-25T01:21:51.118920+0000","last_peered":"2024-04-25T01:21:51.118920+0000","last_clean":"2024-04-25T01:21:51.118920+0000","last_became_active":"2024-04-25T01:21:46.093468+0000","last_became_peered":"2024-04-25T01:21:46.093468+0000","last_unstale":"2024-04-25T01:21:51.118920+0000","last_undegraded":"2024-04-25T01:21:51.118920+0000","last_fullsized":"2024-04-25T01:21:51.118920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119290+0000","last_change":"2024-04-25T01:21:46.096039+0000","last_active":"2024-04-25T01:21:51.119290+0000","last_peered":"2024-04-25T01:21:51.119290+0000","last_clean":"2024-04-25T01:21:51.119290+0000","last_became_active":"2024-04-25T01:21:46.095773+0000","last_became_peered":"2024-04-25T01:21:46.095773+0000","last_unstale":"2024-04-25T01:21:51.119290+0000","last_undegraded":"2024-04-25T01:21:51.119290+0000","last_fullsized":"2024-04-25T01:21:51.119290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472805+0000","last_change":"2024-04-25T01:21:46.106320+0000","last_active":"2024-04-25T01:21:51.472805+0000","last_peered":"2024-04-25T01:21:51.472805+0000","last_clean":"2024-04-25T01:21:51.472805+0000","last_became_active":"2024-04-25T01:21:46.106064+0000","last_became_peered":"2024-04-25T01:21:46.106064+0000","last_unstale":"2024-04-25T01:21:51.472805+0000","last_undegraded":"2024-04-25T01:21:51.472805+0000","last_fullsized":"2024-04-25T01:21:51.472805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:24.344315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471632+0000","last_change":"2024-04-25T01:21:46.093521+0000","last_active":"2024-04-25T01:21:51.471632+0000","last_peered":"2024-04-25T01:21:51.471632+0000","last_clean":"2024-04-25T01:21:51.471632+0000","last_became_active":"2024-04-25T01:21:46.093269+0000","last_became_peered":"2024-04-25T01:21:46.093269+0000","last_unstale":"2024-04-25T01:21:51.471632+0000","last_undegraded":"2024-04-25T01:21:51.471632+0000","last_fullsized":"2024-04-25T01:21:51.471632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472027+0000","last_change":"2024-04-25T01:21:46.102728+0000","last_active":"2024-04-25T01:21:51.472027+0000","last_peered":"2024-04-25T01:21:51.472027+0000","last_clean":"2024-04-25T01:21:51.472027+0000","last_became_active":"2024-04-25T01:21:46.102475+0000","last_became_peered":"2024-04-25T01:21:46.102475+0000","last_unstale":"2024-04-25T01:21:51.472027+0000","last_undegraded":"2024-04-25T01:21:51.472027+0000","last_fullsized":"2024-04-25T01:21:51.472027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472990+0000","last_change":"2024-04-25T01:21:46.101690+0000","last_active":"2024-04-25T01:21:51.472990+0000","last_peered":"2024-04-25T01:21:51.472990+0000","last_clean":"2024-04-25T01:21:51.472990+0000","last_became_active":"2024-04-25T01:21:46.101404+0000","last_became_peered":"2024-04-25T01:21:46.101404+0000","last_unstale":"2024-04-25T01:21:51.472990+0000","last_undegraded":"2024-04-25T01:21:51.472990+0000","last_fullsized":"2024-04-25T01:21:51.472990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472132+0000","last_change":"2024-04-25T01:21:46.103900+0000","last_active":"2024-04-25T01:21:51.472132+0000","last_peered":"2024-04-25T01:21:51.472132+0000","last_clean":"2024-04-25T01:21:51.472132+0000","last_became_active":"2024-04-25T01:21:46.103633+0000","last_became_peered":"2024-04-25T01:21:46.103633+0000","last_unstale":"2024-04-25T01:21:51.472132+0000","last_undegraded":"2024-04-25T01:21:51.472132+0000","last_fullsized":"2024-04-25T01:21:51.472132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:42.961329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472786+0000","last_change":"2024-04-25T01:21:46.102312+0000","last_active":"2024-04-25T01:21:51.472786+0000","last_peered":"2024-04-25T01:21:51.472786+0000","last_clean":"2024-04-25T01:21:51.472786+0000","last_became_active":"2024-04-25T01:21:46.101113+0000","last_became_peered":"2024-04-25T01:21:46.101113+0000","last_unstale":"2024-04-25T01:21:51.472786+0000","last_undegraded":"2024-04-25T01:21:51.472786+0000","last_fullsized":"2024-04-25T01:21:51.472786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:46.038059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473192+0000","last_change":"2024-04-25T01:21:46.098593+0000","last_active":"2024-04-25T01:21:51.473192+0000","last_peered":"2024-04-25T01:21:51.473192+0000","last_clean":"2024-04-25T01:21:51.473192+0000","last_became_active":"2024-04-25T01:21:46.098289+0000","last_became_peered":"2024-04-25T01:21:46.098289+0000","last_unstale":"2024-04-25T01:21:51.473192+0000","last_undegraded":"2024-04-25T01:21:51.473192+0000","last_fullsized":"2024-04-25T01:21:51.473192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473504+0000","last_change":"2024-04-25T01:21:46.093954+0000","last_active":"2024-04-25T01:21:51.473504+0000","last_peered":"2024-04-25T01:21:51.473504+0000","last_clean":"2024-04-25T01:21:51.473504+0000","last_became_active":"2024-04-25T01:21:46.093658+0000","last_became_peered":"2024-04-25T01:21:46.093658+0000","last_unstale":"2024-04-25T01:21:51.473504+0000","last_undegraded":"2024-04-25T01:21:51.473504+0000","last_fullsized":"2024-04-25T01:21:51.473504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472718+0000","last_change":"2024-04-25T01:21:46.108321+0000","last_active":"2024-04-25T01:21:51.472718+0000","last_peered":"2024-04-25T01:21:51.472718+0000","last_clean":"2024-04-25T01:21:51.472718+0000","last_became_active":"2024-04-25T01:21:46.108056+0000","last_became_peered":"2024-04-25T01:21:46.108056+0000","last_unstale":"2024-04-25T01:21:51.472718+0000","last_undegraded":"2024-04-25T01:21:51.472718+0000","last_fullsized":"2024-04-25T01:21:51.472718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:22:08.164 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:22:08.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.313+0000 7f4ebb6ee640 1 Processor -- start 2024-04-25T01:22:08.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.313+0000 7f4ebb6ee640 1 -- start start 2024-04-25T01:22:08.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.314+0000 7f4ebb6ee640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4eb4145590 0x7f4eb40b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.314+0000 7f4ebb6ee640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4eb40b4240 con 0x7f4eb415f9b0 2024-04-25T01:22:08.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.314+0000 7f4ebb6ee640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4eb40b4f60 con 0x7f4eb406e280 2024-04-25T01:22:08.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.314+0000 7f4ebb6ee640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f4eb4160830 con 0x7f4eb4145590 2024-04-25T01:22:08.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.314+0000 7f4eb948a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4eb4145590 0x7f4eb40b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.314+0000 7f4eb8c89640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4eb415f9b0 0x7f4eb40ce110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56390/0 (socket says 172.21.15.77:56390) 2024-04-25T01:22:08.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.314+0000 7f4eb8c89640 1 -- 172.21.15.77:0/3551320512 learned_addr learned my addr 172.21.15.77:0/3551320512 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:08.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.314+0000 7f4eb948a640 1 -- 172.21.15.77:0/3551320512 >> v1:172.21.15.121:6789/0 conn(0x7f4eb406e280 legacy=0x7f4eb40ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.314+0000 7f4eb948a640 1 -- 172.21.15.77:0/3551320512 >> v1:172.21.15.77:6789/0 conn(0x7f4eb415f9b0 legacy=0x7f4eb40ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.314+0000 7f4eb948a640 1 -- 172.21.15.77:0/3551320512 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb40b39a0 con 0x7f4eb4145590 2024-04-25T01:22:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.315+0000 7f4eb948a640 1 --2- 172.21.15.77:0/3551320512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb4145590 0x7f4eb40b4a20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0009d90 tx=0x7f4eb002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1045c9e46ab589e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.315+0000 7f4eabfff640 1 -- 172.21.15.77:0/3551320512 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0007740 con 0x7f4eb4145590 2024-04-25T01:22:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.315+0000 7f4eabfff640 1 -- 172.21.15.77:0/3551320512 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb00078e0 con 0x7f4eb4145590 2024-04-25T01:22:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.315+0000 7f4eabfff640 1 -- 172.21.15.77:0/3551320512 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0007cb0 con 0x7f4eb4145590 2024-04-25T01:22:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.316+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/3551320512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb4145590 msgr2=0x7f4eb40b4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.316+0000 7f4ebb6ee640 1 --2- 172.21.15.77:0/3551320512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb4145590 0x7f4eb40b4a20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0009d90 tx=0x7f4eb002a6a0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.316+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/3551320512 shutdown_connections 2024-04-25T01:22:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.316+0000 7f4ebb6ee640 1 --2- 172.21.15.77:0/3551320512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb4145590 0x7f4eb40b4a20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.316+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/3551320512 >> 172.21.15.77:0/3551320512 conn(0x7f4eb40b1820 msgr2=0x7f4eb40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.316+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/3551320512 shutdown_connections 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.316+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/3551320512 wait complete. 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.317+0000 7f4ebb6ee640 1 Processor -- start 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.317+0000 7f4ebb6ee640 1 -- start start 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4ebb6ee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4eb406e280 0x7f4eb415b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4ebb6ee640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4eb4145590 0x7f4eb413fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4ebb6ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb415f9b0 0x7f4eb41402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4ebb6ee640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb40b3f60 con 0x7f4eb415f9b0 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4ebb6ee640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb406cde0 con 0x7f4eb4145590 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4ebb6ee640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4eb4145970 con 0x7f4eb406e280 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb9c8b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb415f9b0 0x7f4eb41402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb9c8b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb415f9b0 0x7f4eb41402a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56028/0 (socket says 172.21.15.77:56028) 2024-04-25T01:22:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb9c8b640 1 -- 172.21.15.77:0/1380290363 learned_addr learned my addr 172.21.15.77:0/1380290363 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb8c89640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4eb4145590 0x7f4eb413fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb948a640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4eb406e280 0x7f4eb415b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb9c8b640 1 -- 172.21.15.77:0/1380290363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4eb406e280 msgr2=0x7f4eb415b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb9c8b640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4eb406e280 0x7f4eb415b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb9c8b640 1 -- 172.21.15.77:0/1380290363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4eb4145590 msgr2=0x7f4eb413fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb9c8b640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4eb4145590 0x7f4eb413fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb9c8b640 1 -- 172.21.15.77:0/1380290363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb4070dd0 con 0x7f4eb415f9b0 2024-04-25T01:22:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb8c89640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4eb4145590 0x7f4eb413fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb948a640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4eb406e280 0x7f4eb415b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4eb9c8b640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb415f9b0 0x7f4eb41402a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00c410 tx=0x7f4eac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4ea9ffb640 1 -- 172.21.15.77:0/1380290363 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac011790 con 0x7f4eb415f9b0 2024-04-25T01:22:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4ea9ffb640 1 -- 172.21.15.77:0/1380290363 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eac017070 con 0x7f4eb415f9b0 2024-04-25T01:22:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/1380290363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eb40710b0 con 0x7f4eb415f9b0 2024-04-25T01:22:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4ea9ffb640 1 -- 172.21.15.77:0/1380290363 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac011bb0 con 0x7f4eb415f9b0 2024-04-25T01:22:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.319+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/1380290363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eb4071670 con 0x7f4eb415f9b0 2024-04-25T01:22:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.320+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/1380290363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eb4068fb0 con 0x7f4eb415f9b0 2024-04-25T01:22:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.320+0000 7f4ea9ffb640 1 -- 172.21.15.77:0/1380290363 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4eac01c460 con 0x7f4eb415f9b0 2024-04-25T01:22:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.320+0000 7f4ea9ffb640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e9806be30 0x7f4e9806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.320+0000 7f4ea9ffb640 1 -- 172.21.15.77:0/1380290363 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4eac08ad60 con 0x7f4eb415f9b0 2024-04-25T01:22:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.320+0000 7f4eb948a640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e9806be30 0x7f4e9806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.321+0000 7f4eb948a640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e9806be30 0x7f4e9806e2f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0034000 tx=0x7f4eb002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:08.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.324+0000 7f4ea9ffb640 1 -- 172.21.15.77:0/1380290363 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4eac021020 con 0x7f4eb415f9b0 2024-04-25T01:22:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.494+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/1380290363 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4eb406ea50 con 0x7f4e9806be30 2024-04-25T01:22:08.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.501+0000 7f4ea9ffb640 1 -- 172.21.15.77:0/1380290363 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f4eb406ea50 con 0x7f4e9806be30 2024-04-25T01:22:08.486 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:08.488 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:22:08.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/1380290363 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e9806be30 msgr2=0x7f4e9806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e9806be30 0x7f4e9806e2f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0034000 tx=0x7f4eb002f000 comp rx=0 tx=0).stop 2024-04-25T01:22:08.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/1380290363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb415f9b0 msgr2=0x7f4eb41402a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb415f9b0 0x7f4eb41402a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00c410 tx=0x7f4eac00c8d0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/1380290363 shutdown_connections 2024-04-25T01:22:08.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e9806be30 0x7f4e9806e2f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb415f9b0 0x7f4eb41402a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4eb4145590 0x7f4eb413fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 --2- 172.21.15.77:0/1380290363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4eb406e280 0x7f4eb415b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/1380290363 >> 172.21.15.77:0/1380290363 conn(0x7f4eb40b1820 msgr2=0x7f4eb40afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:08.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/1380290363 shutdown_connections 2024-04-25T01:22:08.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.505+0000 7f4ebb6ee640 1 -- 172.21.15.77:0/1380290363 wait complete. 2024-04-25T01:22:08.497 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-25T01:22:07.160553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219400,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755320,"statfs":{"total":773094113280,"available":772869447680,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003046"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471974+0000","last_change":"2024-04-25T01:21:45.082729+0000","last_active":"2024-04-25T01:21:51.471974+0000","last_peered":"2024-04-25T01:21:51.471974+0000","last_clean":"2024-04-25T01:21:51.471974+0000","last_became_active":"2024-04-25T01:21:45.082441+0000","last_became_peered":"2024-04-25T01:21:45.082441+0000","last_unstale":"2024-04-25T01:21:51.471974+0000","last_undegraded":"2024-04-25T01:21:51.471974+0000","last_fullsized":"2024-04-25T01:21:51.471974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:31.228997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472236+0000","last_change":"2024-04-25T01:21:46.093885+0000","last_active":"2024-04-25T01:21:51.472236+0000","last_peered":"2024-04-25T01:21:51.472236+0000","last_clean":"2024-04-25T01:21:51.472236+0000","last_became_active":"2024-04-25T01:21:46.092142+0000","last_became_peered":"2024-04-25T01:21:46.092142+0000","last_unstale":"2024-04-25T01:21:51.472236+0000","last_undegraded":"2024-04-25T01:21:51.472236+0000","last_fullsized":"2024-04-25T01:21:51.472236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471738+0000","last_change":"2024-04-25T01:21:50.187217+0000","last_active":"2024-04-25T01:21:51.471738+0000","last_peered":"2024-04-25T01:21:51.471738+0000","last_clean":"2024-04-25T01:21:51.471738+0000","last_became_active":"2024-04-25T01:21:50.186698+0000","last_became_peered":"2024-04-25T01:21:50.186698+0000","last_unstale":"2024-04-25T01:21:51.471738+0000","last_undegraded":"2024-04-25T01:21:51.471738+0000","last_fullsized":"2024-04-25T01:21:51.471738+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:50.627648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472373+0000","last_change":"2024-04-25T01:21:45.078191+0000","last_active":"2024-04-25T01:21:51.472373+0000","last_peered":"2024-04-25T01:21:51.472373+0000","last_clean":"2024-04-25T01:21:51.472373+0000","last_became_active":"2024-04-25T01:21:45.077949+0000","last_became_peered":"2024-04-25T01:21:45.077949+0000","last_unstale":"2024-04-25T01:21:51.472373+0000","last_undegraded":"2024-04-25T01:21:51.472373+0000","last_fullsized":"2024-04-25T01:21:51.472373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:23.371789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:46.091354+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:46.090827+0000","last_became_peered":"2024-04-25T01:21:46.090827+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.117971+0000","last_change":"2024-04-25T01:21:50.126246+0000","last_active":"2024-04-25T01:21:51.117971+0000","last_peered":"2024-04-25T01:21:51.117971+0000","last_clean":"2024-04-25T01:21:51.117971+0000","last_became_active":"2024-04-25T01:21:50.125823+0000","last_became_peered":"2024-04-25T01:21:50.125823+0000","last_unstale":"2024-04-25T01:21:51.117971+0000","last_undegraded":"2024-04-25T01:21:51.117971+0000","last_fullsized":"2024-04-25T01:21:51.117971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:05.469540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472877+0000","last_change":"2024-04-25T01:21:45.080811+0000","last_active":"2024-04-25T01:21:51.472877+0000","last_peered":"2024-04-25T01:21:51.472877+0000","last_clean":"2024-04-25T01:21:51.472877+0000","last_became_active":"2024-04-25T01:21:45.079902+0000","last_became_peered":"2024-04-25T01:21:45.079902+0000","last_unstale":"2024-04-25T01:21:51.472877+0000","last_undegraded":"2024-04-25T01:21:51.472877+0000","last_fullsized":"2024-04-25T01:21:51.472877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:15.208981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472902+0000","last_change":"2024-04-25T01:21:46.090986+0000","last_active":"2024-04-25T01:21:51.472902+0000","last_peered":"2024-04-25T01:21:51.472902+0000","last_clean":"2024-04-25T01:21:51.472902+0000","last_became_active":"2024-04-25T01:21:46.090715+0000","last_became_peered":"2024-04-25T01:21:46.090715+0000","last_unstale":"2024-04-25T01:21:51.472902+0000","last_undegraded":"2024-04-25T01:21:51.472902+0000","last_fullsized":"2024-04-25T01:21:51.472902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:27.583834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118940+0000","last_change":"2024-04-25T01:21:50.128507+0000","last_active":"2024-04-25T01:21:51.118940+0000","last_peered":"2024-04-25T01:21:51.118940+0000","last_clean":"2024-04-25T01:21:51.118940+0000","last_became_active":"2024-04-25T01:21:50.128116+0000","last_became_peered":"2024-04-25T01:21:50.128116+0000","last_unstale":"2024-04-25T01:21:51.118940+0000","last_undegraded":"2024-04-25T01:21:51.118940+0000","last_fullsized":"2024-04-25T01:21:51.118940+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.077404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472303+0000","last_change":"2024-04-25T01:21:45.080401+0000","last_active":"2024-04-25T01:21:51.472303+0000","last_peered":"2024-04-25T01:21:51.472303+0000","last_clean":"2024-04-25T01:21:51.472303+0000","last_became_active":"2024-04-25T01:21:45.079961+0000","last_became_peered":"2024-04-25T01:21:45.079961+0000","last_unstale":"2024-04-25T01:21:51.472303+0000","last_undegraded":"2024-04-25T01:21:51.472303+0000","last_fullsized":"2024-04-25T01:21:51.472303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473268+0000","last_change":"2024-04-25T01:21:46.090524+0000","last_active":"2024-04-25T01:21:51.473268+0000","last_peered":"2024-04-25T01:21:51.473268+0000","last_clean":"2024-04-25T01:21:51.473268+0000","last_became_active":"2024-04-25T01:21:46.089869+0000","last_became_peered":"2024-04-25T01:21:46.089869+0000","last_unstale":"2024-04-25T01:21:51.473268+0000","last_undegraded":"2024-04-25T01:21:51.473268+0000","last_fullsized":"2024-04-25T01:21:51.473268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:24.344315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118039+0000","last_change":"2024-04-25T01:21:50.118075+0000","last_active":"2024-04-25T01:21:51.118039+0000","last_peered":"2024-04-25T01:21:51.118039+0000","last_clean":"2024-04-25T01:21:51.118039+0000","last_became_active":"2024-04-25T01:21:50.117657+0000","last_became_peered":"2024-04-25T01:21:50.117657+0000","last_unstale":"2024-04-25T01:21:51.118039+0000","last_undegraded":"2024-04-25T01:21:51.118039+0000","last_fullsized":"2024-04-25T01:21:51.118039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:29.510529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118231+0000","last_change":"2024-04-25T01:21:46.093011+0000","last_active":"2024-04-25T01:21:51.118231+0000","last_peered":"2024-04-25T01:21:51.118231+0000","last_clean":"2024-04-25T01:21:51.118231+0000","last_became_active":"2024-04-25T01:21:46.092710+0000","last_became_peered":"2024-04-25T01:21:46.092710+0000","last_unstale":"2024-04-25T01:21:51.118231+0000","last_undegraded":"2024-04-25T01:21:51.118231+0000","last_fullsized":"2024-04-25T01:21:51.118231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472603+0000","last_change":"2024-04-25T01:21:45.080010+0000","last_active":"2024-04-25T01:21:51.472603+0000","last_peered":"2024-04-25T01:21:51.472603+0000","last_clean":"2024-04-25T01:21:51.472603+0000","last_became_active":"2024-04-25T01:21:45.079732+0000","last_became_peered":"2024-04-25T01:21:45.079732+0000","last_unstale":"2024-04-25T01:21:51.472603+0000","last_undegraded":"2024-04-25T01:21:51.472603+0000","last_fullsized":"2024-04-25T01:21:51.472603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119330+0000","last_change":"2024-04-25T01:21:50.121929+0000","last_active":"2024-04-25T01:21:51.119330+0000","last_peered":"2024-04-25T01:21:51.119330+0000","last_clean":"2024-04-25T01:21:51.119330+0000","last_became_active":"2024-04-25T01:21:50.121288+0000","last_became_peered":"2024-04-25T01:21:50.121288+0000","last_unstale":"2024-04-25T01:21:51.119330+0000","last_undegraded":"2024-04-25T01:21:51.119330+0000","last_fullsized":"2024-04-25T01:21:51.119330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473407+0000","last_change":"2024-04-25T01:21:46.082691+0000","last_active":"2024-04-25T01:21:51.473407+0000","last_peered":"2024-04-25T01:21:51.473407+0000","last_clean":"2024-04-25T01:21:51.473407+0000","last_became_active":"2024-04-25T01:21:46.082266+0000","last_became_peered":"2024-04-25T01:21:46.082266+0000","last_unstale":"2024-04-25T01:21:51.473407+0000","last_undegraded":"2024-04-25T01:21:51.473407+0000","last_fullsized":"2024-04-25T01:21:51.473407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:22.971333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473925+0000","last_change":"2024-04-25T01:21:45.076031+0000","last_active":"2024-04-25T01:21:51.473925+0000","last_peered":"2024-04-25T01:21:51.473925+0000","last_clean":"2024-04-25T01:21:51.473925+0000","last_became_active":"2024-04-25T01:21:45.075769+0000","last_became_peered":"2024-04-25T01:21:45.075769+0000","last_unstale":"2024-04-25T01:21:51.473925+0000","last_undegraded":"2024-04-25T01:21:51.473925+0000","last_fullsized":"2024-04-25T01:21:51.473925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118699+0000","last_change":"2024-04-25T01:21:50.122266+0000","last_active":"2024-04-25T01:21:51.118699+0000","last_peered":"2024-04-25T01:21:51.118699+0000","last_clean":"2024-04-25T01:21:51.118699+0000","last_became_active":"2024-04-25T01:21:50.121879+0000","last_became_peered":"2024-04-25T01:21:50.121879+0000","last_unstale":"2024-04-25T01:21:51.118699+0000","last_undegraded":"2024-04-25T01:21:51.118699+0000","last_fullsized":"2024-04-25T01:21:51.118699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:51.825101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471956+0000","last_change":"2024-04-25T01:21:46.093175+0000","last_active":"2024-04-25T01:21:51.471956+0000","last_peered":"2024-04-25T01:21:51.471956+0000","last_clean":"2024-04-25T01:21:51.471956+0000","last_became_active":"2024-04-25T01:21:46.092327+0000","last_became_peered":"2024-04-25T01:21:46.092327+0000","last_unstale":"2024-04-25T01:21:51.471956+0000","last_undegraded":"2024-04-25T01:21:51.471956+0000","last_fullsized":"2024-04-25T01:21:51.471956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473162+0000","last_change":"2024-04-25T01:21:45.077148+0000","last_active":"2024-04-25T01:21:51.473162+0000","last_peered":"2024-04-25T01:21:51.473162+0000","last_clean":"2024-04-25T01:21:51.473162+0000","last_became_active":"2024-04-25T01:21:45.076858+0000","last_became_peered":"2024-04-25T01:21:45.076858+0000","last_unstale":"2024-04-25T01:21:51.473162+0000","last_undegraded":"2024-04-25T01:21:51.473162+0000","last_fullsized":"2024-04-25T01:21:51.473162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:15.208981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473795+0000","last_change":"2024-04-25T01:21:50.182249+0000","last_active":"2024-04-25T01:21:51.473795+0000","last_peered":"2024-04-25T01:21:51.473795+0000","last_clean":"2024-04-25T01:21:51.473795+0000","last_became_active":"2024-04-25T01:21:50.181860+0000","last_became_peered":"2024-04-25T01:21:50.181860+0000","last_unstale":"2024-04-25T01:21:51.473795+0000","last_undegraded":"2024-04-25T01:21:51.473795+0000","last_fullsized":"2024-04-25T01:21:51.473795+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:38.806941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119159+0000","last_change":"2024-04-25T01:21:46.085766+0000","last_active":"2024-04-25T01:21:51.119159+0000","last_peered":"2024-04-25T01:21:51.119159+0000","last_clean":"2024-04-25T01:21:51.119159+0000","last_became_active":"2024-04-25T01:21:46.085479+0000","last_became_peered":"2024-04-25T01:21:46.085479+0000","last_unstale":"2024-04-25T01:21:51.119159+0000","last_undegraded":"2024-04-25T01:21:51.119159+0000","last_fullsized":"2024-04-25T01:21:51.119159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:02.493076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:45.085085+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:45.084796+0000","last_became_peered":"2024-04-25T01:21:45.084796+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:25.950316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118843+0000","last_change":"2024-04-25T01:21:50.123379+0000","last_active":"2024-04-25T01:21:51.118843+0000","last_peered":"2024-04-25T01:21:51.118843+0000","last_clean":"2024-04-25T01:21:51.118843+0000","last_became_active":"2024-04-25T01:21:50.123057+0000","last_became_peered":"2024-04-25T01:21:50.123057+0000","last_unstale":"2024-04-25T01:21:51.118843+0000","last_undegraded":"2024-04-25T01:21:51.118843+0000","last_fullsized":"2024-04-25T01:21:51.118843+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:13.828811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472820+0000","last_change":"2024-04-25T01:21:46.090075+0000","last_active":"2024-04-25T01:21:51.472820+0000","last_peered":"2024-04-25T01:21:51.472820+0000","last_clean":"2024-04-25T01:21:51.472820+0000","last_became_active":"2024-04-25T01:21:46.089800+0000","last_became_peered":"2024-04-25T01:21:46.089800+0000","last_unstale":"2024-04-25T01:21:51.472820+0000","last_undegraded":"2024-04-25T01:21:51.472820+0000","last_fullsized":"2024-04-25T01:21:51.472820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119254+0000","last_change":"2024-04-25T01:21:45.080102+0000","last_active":"2024-04-25T01:21:51.119254+0000","last_peered":"2024-04-25T01:21:51.119254+0000","last_clean":"2024-04-25T01:21:51.119254+0000","last_became_active":"2024-04-25T01:21:45.079786+0000","last_became_peered":"2024-04-25T01:21:45.079786+0000","last_unstale":"2024-04-25T01:21:51.119254+0000","last_undegraded":"2024-04-25T01:21:51.119254+0000","last_fullsized":"2024-04-25T01:21:51.119254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:51.235948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118619+0000","last_change":"2024-04-25T01:21:50.185727+0000","last_active":"2024-04-25T01:21:51.118619+0000","last_peered":"2024-04-25T01:21:51.118619+0000","last_clean":"2024-04-25T01:21:51.118619+0000","last_became_active":"2024-04-25T01:21:50.184395+0000","last_became_peered":"2024-04-25T01:21:50.184395+0000","last_unstale":"2024-04-25T01:21:51.118619+0000","last_undegraded":"2024-04-25T01:21:51.118619+0000","last_fullsized":"2024-04-25T01:21:51.118619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:05.257957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472627+0000","last_change":"2024-04-25T01:21:46.099799+0000","last_active":"2024-04-25T01:21:51.472627+0000","last_peered":"2024-04-25T01:21:51.472627+0000","last_clean":"2024-04-25T01:21:51.472627+0000","last_became_active":"2024-04-25T01:21:46.099506+0000","last_became_peered":"2024-04-25T01:21:46.099506+0000","last_unstale":"2024-04-25T01:21:51.472627+0000","last_undegraded":"2024-04-25T01:21:51.472627+0000","last_fullsized":"2024-04-25T01:21:51.472627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473591+0000","last_change":"2024-04-25T01:21:45.079966+0000","last_active":"2024-04-25T01:21:51.473591+0000","last_peered":"2024-04-25T01:21:51.473591+0000","last_clean":"2024-04-25T01:21:51.473591+0000","last_became_active":"2024-04-25T01:21:45.079689+0000","last_became_peered":"2024-04-25T01:21:45.079689+0000","last_unstale":"2024-04-25T01:21:51.473591+0000","last_undegraded":"2024-04-25T01:21:51.473591+0000","last_fullsized":"2024-04-25T01:21:51.473591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471654+0000","last_change":"2024-04-25T01:21:50.184865+0000","last_active":"2024-04-25T01:21:51.471654+0000","last_peered":"2024-04-25T01:21:51.471654+0000","last_clean":"2024-04-25T01:21:51.471654+0000","last_became_active":"2024-04-25T01:21:50.184466+0000","last_became_peered":"2024-04-25T01:21:50.184466+0000","last_unstale":"2024-04-25T01:21:51.471654+0000","last_undegraded":"2024-04-25T01:21:51.471654+0000","last_fullsized":"2024-04-25T01:21:51.471654+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:42.606728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473876+0000","last_change":"2024-04-25T01:21:46.100915+0000","last_active":"2024-04-25T01:21:51.473876+0000","last_peered":"2024-04-25T01:21:51.473876+0000","last_clean":"2024-04-25T01:21:51.473876+0000","last_became_active":"2024-04-25T01:21:46.100641+0000","last_became_peered":"2024-04-25T01:21:46.100641+0000","last_unstale":"2024-04-25T01:21:51.473876+0000","last_undegraded":"2024-04-25T01:21:51.473876+0000","last_fullsized":"2024-04-25T01:21:51.473876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472835+0000","last_change":"2024-04-25T01:21:45.085103+0000","last_active":"2024-04-25T01:21:51.472835+0000","last_peered":"2024-04-25T01:21:51.472835+0000","last_clean":"2024-04-25T01:21:51.472835+0000","last_became_active":"2024-04-25T01:21:45.084805+0000","last_became_peered":"2024-04-25T01:21:45.084805+0000","last_unstale":"2024-04-25T01:21:51.472835+0000","last_undegraded":"2024-04-25T01:21:51.472835+0000","last_fullsized":"2024-04-25T01:21:51.472835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:56.444547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471942+0000","last_change":"2024-04-25T01:21:50.184815+0000","last_active":"2024-04-25T01:21:51.471942+0000","last_peered":"2024-04-25T01:21:51.471942+0000","last_clean":"2024-04-25T01:21:51.471942+0000","last_became_active":"2024-04-25T01:21:50.184441+0000","last_became_peered":"2024-04-25T01:21:50.184441+0000","last_unstale":"2024-04-25T01:21:51.471942+0000","last_undegraded":"2024-04-25T01:21:51.471942+0000","last_fullsized":"2024-04-25T01:21:51.471942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473364+0000","last_change":"2024-04-25T01:21:46.096428+0000","last_active":"2024-04-25T01:21:51.473364+0000","last_peered":"2024-04-25T01:21:51.473364+0000","last_clean":"2024-04-25T01:21:51.473364+0000","last_became_active":"2024-04-25T01:21:46.096151+0000","last_became_peered":"2024-04-25T01:21:46.096151+0000","last_unstale":"2024-04-25T01:21:51.473364+0000","last_undegraded":"2024-04-25T01:21:51.473364+0000","last_fullsized":"2024-04-25T01:21:51.473364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:42.961329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473960+0000","last_change":"2024-04-25T01:21:45.080814+0000","last_active":"2024-04-25T01:21:51.473960+0000","last_peered":"2024-04-25T01:21:51.473960+0000","last_clean":"2024-04-25T01:21:51.473960+0000","last_became_active":"2024-04-25T01:21:45.080505+0000","last_became_peered":"2024-04-25T01:21:45.080505+0000","last_unstale":"2024-04-25T01:21:51.473960+0000","last_undegraded":"2024-04-25T01:21:51.473960+0000","last_fullsized":"2024-04-25T01:21:51.473960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473549+0000","last_change":"2024-04-25T01:21:50.129319+0000","last_active":"2024-04-25T01:21:51.473549+0000","last_peered":"2024-04-25T01:21:51.473549+0000","last_clean":"2024-04-25T01:21:51.473549+0000","last_became_active":"2024-04-25T01:21:50.128797+0000","last_became_peered":"2024-04-25T01:21:50.128797+0000","last_unstale":"2024-04-25T01:21:51.473549+0000","last_undegraded":"2024-04-25T01:21:51.473549+0000","last_fullsized":"2024-04-25T01:21:51.473549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:31.623179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119209+0000","last_change":"2024-04-25T01:21:46.079764+0000","last_active":"2024-04-25T01:21:51.119209+0000","last_peered":"2024-04-25T01:21:51.119209+0000","last_clean":"2024-04-25T01:21:51.119209+0000","last_became_active":"2024-04-25T01:21:46.079475+0000","last_became_peered":"2024-04-25T01:21:46.079475+0000","last_unstale":"2024-04-25T01:21:51.119209+0000","last_undegraded":"2024-04-25T01:21:51.119209+0000","last_fullsized":"2024-04-25T01:21:51.119209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:21.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473306+0000","last_change":"2024-04-25T01:21:45.075554+0000","last_active":"2024-04-25T01:21:51.473306+0000","last_peered":"2024-04-25T01:21:51.473306+0000","last_clean":"2024-04-25T01:21:51.473306+0000","last_became_active":"2024-04-25T01:21:45.075256+0000","last_became_peered":"2024-04-25T01:21:45.075256+0000","last_unstale":"2024-04-25T01:21:51.473306+0000","last_undegraded":"2024-04-25T01:21:51.473306+0000","last_fullsized":"2024-04-25T01:21:51.473306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473829+0000","last_change":"2024-04-25T01:21:50.131544+0000","last_active":"2024-04-25T01:21:51.473829+0000","last_peered":"2024-04-25T01:21:51.473829+0000","last_clean":"2024-04-25T01:21:51.473829+0000","last_became_active":"2024-04-25T01:21:50.131125+0000","last_became_peered":"2024-04-25T01:21:50.131125+0000","last_unstale":"2024-04-25T01:21:51.473829+0000","last_undegraded":"2024-04-25T01:21:51.473829+0000","last_fullsized":"2024-04-25T01:21:51.473829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:00.042952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472863+0000","last_change":"2024-04-25T01:21:46.094965+0000","last_active":"2024-04-25T01:21:51.472863+0000","last_peered":"2024-04-25T01:21:51.472863+0000","last_clean":"2024-04-25T01:21:51.472863+0000","last_became_active":"2024-04-25T01:21:46.094515+0000","last_became_peered":"2024-04-25T01:21:46.094515+0000","last_unstale":"2024-04-25T01:21:51.472863+0000","last_undegraded":"2024-04-25T01:21:51.472863+0000","last_fullsized":"2024-04-25T01:21:51.472863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119246+0000","last_change":"2024-04-25T01:21:45.075901+0000","last_active":"2024-04-25T01:21:51.119246+0000","last_peered":"2024-04-25T01:21:51.119246+0000","last_clean":"2024-04-25T01:21:51.119246+0000","last_became_active":"2024-04-25T01:21:45.075635+0000","last_became_peered":"2024-04-25T01:21:45.075635+0000","last_unstale":"2024-04-25T01:21:51.119246+0000","last_undegraded":"2024-04-25T01:21:51.119246+0000","last_fullsized":"2024-04-25T01:21:51.119246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473778+0000","last_change":"2024-04-25T01:21:50.129249+0000","last_active":"2024-04-25T01:21:51.473778+0000","last_peered":"2024-04-25T01:21:51.473778+0000","last_clean":"2024-04-25T01:21:51.473778+0000","last_became_active":"2024-04-25T01:21:50.128867+0000","last_became_peered":"2024-04-25T01:21:50.128867+0000","last_unstale":"2024-04-25T01:21:51.473778+0000","last_undegraded":"2024-04-25T01:21:51.473778+0000","last_fullsized":"2024-04-25T01:21:51.473778+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:50.627648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471590+0000","last_change":"2024-04-25T01:21:46.078894+0000","last_active":"2024-04-25T01:21:51.471590+0000","last_peered":"2024-04-25T01:21:51.471590+0000","last_clean":"2024-04-25T01:21:51.471590+0000","last_became_active":"2024-04-25T01:21:46.078598+0000","last_became_peered":"2024-04-25T01:21:46.078598+0000","last_unstale":"2024-04-25T01:21:51.471590+0000","last_undegraded":"2024-04-25T01:21:51.471590+0000","last_fullsized":"2024-04-25T01:21:51.471590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:57.449234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471807+0000","last_change":"2024-04-25T01:21:45.080835+0000","last_active":"2024-04-25T01:21:51.471807+0000","last_peered":"2024-04-25T01:21:51.471807+0000","last_clean":"2024-04-25T01:21:51.471807+0000","last_became_active":"2024-04-25T01:21:45.080544+0000","last_became_peered":"2024-04-25T01:21:45.080544+0000","last_unstale":"2024-04-25T01:21:51.471807+0000","last_undegraded":"2024-04-25T01:21:51.471807+0000","last_fullsized":"2024-04-25T01:21:51.471807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472752+0000","last_change":"2024-04-25T01:21:50.182264+0000","last_active":"2024-04-25T01:21:51.472752+0000","last_peered":"2024-04-25T01:21:51.472752+0000","last_clean":"2024-04-25T01:21:51.472752+0000","last_became_active":"2024-04-25T01:21:50.181854+0000","last_became_peered":"2024-04-25T01:21:50.181854+0000","last_unstale":"2024-04-25T01:21:51.472752+0000","last_undegraded":"2024-04-25T01:21:51.472752+0000","last_fullsized":"2024-04-25T01:21:51.472752+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:28.383661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:46.102152+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:46.101917+0000","last_became_peered":"2024-04-25T01:21:46.101917+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473111+0000","last_change":"2024-04-25T01:21:45.070971+0000","last_active":"2024-04-25T01:21:51.473111+0000","last_peered":"2024-04-25T01:21:51.473111+0000","last_clean":"2024-04-25T01:21:51.473111+0000","last_became_active":"2024-04-25T01:21:45.070626+0000","last_became_peered":"2024-04-25T01:21:45.070626+0000","last_unstale":"2024-04-25T01:21:51.473111+0000","last_undegraded":"2024-04-25T01:21:51.473111+0000","last_fullsized":"2024-04-25T01:21:51.473111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471866+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:21:51.471866+0000","last_peered":"2024-04-25T01:21:51.471866+0000","last_clean":"2024-04-25T01:21:51.471866+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:21:51.471866+0000","last_undegraded":"2024-04-25T01:21:51.471866+0000","last_fullsized":"2024-04-25T01:21:51.471866+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118918+0000","last_change":"2024-04-25T01:21:50.185591+0000","last_active":"2024-04-25T01:21:51.118918+0000","last_peered":"2024-04-25T01:21:51.118918+0000","last_clean":"2024-04-25T01:21:51.118918+0000","last_became_active":"2024-04-25T01:21:50.185206+0000","last_became_peered":"2024-04-25T01:21:50.185206+0000","last_unstale":"2024-04-25T01:21:51.118918+0000","last_undegraded":"2024-04-25T01:21:51.118918+0000","last_fullsized":"2024-04-25T01:21:51.118918+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.077404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472462+0000","last_change":"2024-04-25T01:21:46.104388+0000","last_active":"2024-04-25T01:21:51.472462+0000","last_peered":"2024-04-25T01:21:51.472462+0000","last_clean":"2024-04-25T01:21:51.472462+0000","last_became_active":"2024-04-25T01:21:46.103864+0000","last_became_peered":"2024-04-25T01:21:46.103864+0000","last_unstale":"2024-04-25T01:21:51.472462+0000","last_undegraded":"2024-04-25T01:21:51.472462+0000","last_fullsized":"2024-04-25T01:21:51.472462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:46.038059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119490+0000","last_change":"2024-04-25T01:21:45.078985+0000","last_active":"2024-04-25T01:21:51.119490+0000","last_peered":"2024-04-25T01:21:51.119490+0000","last_clean":"2024-04-25T01:21:51.119490+0000","last_became_active":"2024-04-25T01:21:45.078782+0000","last_became_peered":"2024-04-25T01:21:45.078782+0000","last_unstale":"2024-04-25T01:21:51.119490+0000","last_undegraded":"2024-04-25T01:21:51.119490+0000","last_fullsized":"2024-04-25T01:21:51.119490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:51.235948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118175+0000","last_change":"2024-04-25T01:21:50.182278+0000","last_active":"2024-04-25T01:21:51.118175+0000","last_peered":"2024-04-25T01:21:51.118175+0000","last_clean":"2024-04-25T01:21:51.118175+0000","last_became_active":"2024-04-25T01:21:50.181838+0000","last_became_peered":"2024-04-25T01:21:50.181838+0000","last_unstale":"2024-04-25T01:21:51.118175+0000","last_undegraded":"2024-04-25T01:21:51.118175+0000","last_fullsized":"2024-04-25T01:21:51.118175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:13.828811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473013+0000","last_change":"2024-04-25T01:21:46.096433+0000","last_active":"2024-04-25T01:21:51.473013+0000","last_peered":"2024-04-25T01:21:51.473013+0000","last_clean":"2024-04-25T01:21:51.473013+0000","last_became_active":"2024-04-25T01:21:46.096197+0000","last_became_peered":"2024-04-25T01:21:46.096197+0000","last_unstale":"2024-04-25T01:21:51.473013+0000","last_undegraded":"2024-04-25T01:21:51.473013+0000","last_fullsized":"2024-04-25T01:21:51.473013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:38.567383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473546+0000","last_change":"2024-04-25T01:21:45.072096+0000","last_active":"2024-04-25T01:21:51.473546+0000","last_peered":"2024-04-25T01:21:51.473546+0000","last_clean":"2024-04-25T01:21:51.473546+0000","last_became_active":"2024-04-25T01:21:45.071811+0000","last_became_peered":"2024-04-25T01:21:45.071811+0000","last_unstale":"2024-04-25T01:21:51.473546+0000","last_undegraded":"2024-04-25T01:21:51.473546+0000","last_fullsized":"2024-04-25T01:21:51.473546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472183+0000","last_change":"2024-04-25T01:21:50.123158+0000","last_active":"2024-04-25T01:21:51.472183+0000","last_peered":"2024-04-25T01:21:51.472183+0000","last_clean":"2024-04-25T01:21:51.472183+0000","last_became_active":"2024-04-25T01:21:50.122275+0000","last_became_peered":"2024-04-25T01:21:50.122275+0000","last_unstale":"2024-04-25T01:21:51.472183+0000","last_undegraded":"2024-04-25T01:21:51.472183+0000","last_fullsized":"2024-04-25T01:21:51.472183+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:49.064038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472847+0000","last_change":"2024-04-25T01:21:46.105159+0000","last_active":"2024-04-25T01:21:51.472847+0000","last_peered":"2024-04-25T01:21:51.472847+0000","last_clean":"2024-04-25T01:21:51.472847+0000","last_became_active":"2024-04-25T01:21:46.104876+0000","last_became_peered":"2024-04-25T01:21:46.104876+0000","last_unstale":"2024-04-25T01:21:51.472847+0000","last_undegraded":"2024-04-25T01:21:51.472847+0000","last_fullsized":"2024-04-25T01:21:51.472847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:38.567383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119570+0000","last_change":"2024-04-25T01:21:45.076578+0000","last_active":"2024-04-25T01:21:51.119570+0000","last_peered":"2024-04-25T01:21:51.119570+0000","last_clean":"2024-04-25T01:21:51.119570+0000","last_became_active":"2024-04-25T01:21:45.076022+0000","last_became_peered":"2024-04-25T01:21:45.076022+0000","last_unstale":"2024-04-25T01:21:51.119570+0000","last_undegraded":"2024-04-25T01:21:51.119570+0000","last_fullsized":"2024-04-25T01:21:51.119570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471952+0000","last_change":"2024-04-25T01:21:50.182219+0000","last_active":"2024-04-25T01:21:51.471952+0000","last_peered":"2024-04-25T01:21:51.471952+0000","last_clean":"2024-04-25T01:21:51.471952+0000","last_became_active":"2024-04-25T01:21:50.181820+0000","last_became_peered":"2024-04-25T01:21:50.181820+0000","last_unstale":"2024-04-25T01:21:51.471952+0000","last_undegraded":"2024-04-25T01:21:51.471952+0000","last_fullsized":"2024-04-25T01:21:51.471952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:20.747339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472586+0000","last_change":"2024-04-25T01:21:46.100017+0000","last_active":"2024-04-25T01:21:51.472586+0000","last_peered":"2024-04-25T01:21:51.472586+0000","last_clean":"2024-04-25T01:21:51.472586+0000","last_became_active":"2024-04-25T01:21:46.099745+0000","last_became_peered":"2024-04-25T01:21:46.099745+0000","last_unstale":"2024-04-25T01:21:51.472586+0000","last_undegraded":"2024-04-25T01:21:51.472586+0000","last_fullsized":"2024-04-25T01:21:51.472586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:30.138160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118932+0000","last_change":"2024-04-25T01:21:45.081659+0000","last_active":"2024-04-25T01:21:51.118932+0000","last_peered":"2024-04-25T01:21:51.118932+0000","last_clean":"2024-04-25T01:21:51.118932+0000","last_became_active":"2024-04-25T01:21:45.081320+0000","last_became_peered":"2024-04-25T01:21:45.081320+0000","last_unstale":"2024-04-25T01:21:51.118932+0000","last_undegraded":"2024-04-25T01:21:51.118932+0000","last_fullsized":"2024-04-25T01:21:51.118932+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:11:25.817031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118175+0000","last_change":"2024-04-25T01:21:50.122840+0000","last_active":"2024-04-25T01:21:51.118175+0000","last_peered":"2024-04-25T01:21:51.118175+0000","last_clean":"2024-04-25T01:21:51.118175+0000","last_became_active":"2024-04-25T01:21:50.122184+0000","last_became_peered":"2024-04-25T01:21:50.122184+0000","last_unstale":"2024-04-25T01:21:51.118175+0000","last_undegraded":"2024-04-25T01:21:51.118175+0000","last_fullsized":"2024-04-25T01:21:51.118175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:27.010679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472223+0000","last_change":"2024-04-25T01:21:46.105988+0000","last_active":"2024-04-25T01:21:51.472223+0000","last_peered":"2024-04-25T01:21:51.472223+0000","last_clean":"2024-04-25T01:21:51.472223+0000","last_became_active":"2024-04-25T01:21:46.105479+0000","last_became_peered":"2024-04-25T01:21:46.105479+0000","last_unstale":"2024-04-25T01:21:51.472223+0000","last_undegraded":"2024-04-25T01:21:51.472223+0000","last_fullsized":"2024-04-25T01:21:51.472223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:45.024693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119108+0000","last_change":"2024-04-25T01:21:45.074194+0000","last_active":"2024-04-25T01:21:51.119108+0000","last_peered":"2024-04-25T01:21:51.119108+0000","last_clean":"2024-04-25T01:21:51.119108+0000","last_became_active":"2024-04-25T01:21:45.073923+0000","last_became_peered":"2024-04-25T01:21:45.073923+0000","last_unstale":"2024-04-25T01:21:51.119108+0000","last_undegraded":"2024-04-25T01:21:51.119108+0000","last_fullsized":"2024-04-25T01:21:51.119108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472699+0000","last_change":"2024-04-25T01:21:50.126685+0000","last_active":"2024-04-25T01:21:51.472699+0000","last_peered":"2024-04-25T01:21:51.472699+0000","last_clean":"2024-04-25T01:21:51.472699+0000","last_became_active":"2024-04-25T01:21:50.126183+0000","last_became_peered":"2024-04-25T01:21:50.126183+0000","last_unstale":"2024-04-25T01:21:51.472699+0000","last_undegraded":"2024-04-25T01:21:51.472699+0000","last_fullsized":"2024-04-25T01:21:51.472699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.177506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118469+0000","last_change":"2024-04-25T01:21:46.096531+0000","last_active":"2024-04-25T01:21:51.118469+0000","last_peered":"2024-04-25T01:21:51.118469+0000","last_clean":"2024-04-25T01:21:51.118469+0000","last_became_active":"2024-04-25T01:21:46.096250+0000","last_became_peered":"2024-04-25T01:21:46.096250+0000","last_unstale":"2024-04-25T01:21:51.118469+0000","last_undegraded":"2024-04-25T01:21:51.118469+0000","last_fullsized":"2024-04-25T01:21:51.118469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:55.076328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.117985+0000","last_change":"2024-04-25T01:21:45.079574+0000","last_active":"2024-04-25T01:21:51.117985+0000","last_peered":"2024-04-25T01:21:51.117985+0000","last_clean":"2024-04-25T01:21:51.117985+0000","last_became_active":"2024-04-25T01:21:45.079306+0000","last_became_peered":"2024-04-25T01:21:45.079306+0000","last_unstale":"2024-04-25T01:21:51.117985+0000","last_undegraded":"2024-04-25T01:21:51.117985+0000","last_fullsized":"2024-04-25T01:21:51.117985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472488+0000","last_change":"2024-04-25T01:21:50.187252+0000","last_active":"2024-04-25T01:21:51.472488+0000","last_peered":"2024-04-25T01:21:51.472488+0000","last_clean":"2024-04-25T01:21:51.472488+0000","last_became_active":"2024-04-25T01:21:50.186509+0000","last_became_peered":"2024-04-25T01:21:50.186509+0000","last_unstale":"2024-04-25T01:21:51.472488+0000","last_undegraded":"2024-04-25T01:21:51.472488+0000","last_fullsized":"2024-04-25T01:21:51.472488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:15.374624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472403+0000","last_change":"2024-04-25T01:21:46.104318+0000","last_active":"2024-04-25T01:21:51.472403+0000","last_peered":"2024-04-25T01:21:51.472403+0000","last_clean":"2024-04-25T01:21:51.472403+0000","last_became_active":"2024-04-25T01:21:46.103937+0000","last_became_peered":"2024-04-25T01:21:46.103937+0000","last_unstale":"2024-04-25T01:21:51.472403+0000","last_undegraded":"2024-04-25T01:21:51.472403+0000","last_fullsized":"2024-04-25T01:21:51.472403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:27.802903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473263+0000","last_change":"2024-04-25T01:21:45.076486+0000","last_active":"2024-04-25T01:21:51.473263+0000","last_peered":"2024-04-25T01:21:51.473263+0000","last_clean":"2024-04-25T01:21:51.473263+0000","last_became_active":"2024-04-25T01:21:45.076164+0000","last_became_peered":"2024-04-25T01:21:45.076164+0000","last_unstale":"2024-04-25T01:21:51.473263+0000","last_undegraded":"2024-04-25T01:21:51.473263+0000","last_fullsized":"2024-04-25T01:21:51.473263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472905+0000","last_change":"2024-04-25T01:21:50.182316+0000","last_active":"2024-04-25T01:21:51.472905+0000","last_peered":"2024-04-25T01:21:51.472905+0000","last_clean":"2024-04-25T01:21:51.472905+0000","last_became_active":"2024-04-25T01:21:50.181918+0000","last_became_peered":"2024-04-25T01:21:50.181918+0000","last_unstale":"2024-04-25T01:21:51.472905+0000","last_undegraded":"2024-04-25T01:21:51.472905+0000","last_fullsized":"2024-04-25T01:21:51.472905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:37.041783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472461+0000","last_change":"2024-04-25T01:21:46.100833+0000","last_active":"2024-04-25T01:21:51.472461+0000","last_peered":"2024-04-25T01:21:51.472461+0000","last_clean":"2024-04-25T01:21:51.472461+0000","last_became_active":"2024-04-25T01:21:46.100591+0000","last_became_peered":"2024-04-25T01:21:46.100591+0000","last_unstale":"2024-04-25T01:21:51.472461+0000","last_undegraded":"2024-04-25T01:21:51.472461+0000","last_fullsized":"2024-04-25T01:21:51.472461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:36.537071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472415+0000","last_change":"2024-04-25T01:21:45.078469+0000","last_active":"2024-04-25T01:21:51.472415+0000","last_peered":"2024-04-25T01:21:51.472415+0000","last_clean":"2024-04-25T01:21:51.472415+0000","last_became_active":"2024-04-25T01:21:45.078231+0000","last_became_peered":"2024-04-25T01:21:45.078231+0000","last_unstale":"2024-04-25T01:21:51.472415+0000","last_undegraded":"2024-04-25T01:21:51.472415+0000","last_fullsized":"2024-04-25T01:21:51.472415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:23.371789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472786+0000","last_change":"2024-04-25T01:21:50.185207+0000","last_active":"2024-04-25T01:21:51.472786+0000","last_peered":"2024-04-25T01:21:51.472786+0000","last_clean":"2024-04-25T01:21:51.472786+0000","last_became_active":"2024-04-25T01:21:50.184793+0000","last_became_peered":"2024-04-25T01:21:50.184793+0000","last_unstale":"2024-04-25T01:21:51.472786+0000","last_undegraded":"2024-04-25T01:21:51.472786+0000","last_fullsized":"2024-04-25T01:21:51.472786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:15.374624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472438+0000","last_change":"2024-04-25T01:21:46.104460+0000","last_active":"2024-04-25T01:21:51.472438+0000","last_peered":"2024-04-25T01:21:51.472438+0000","last_clean":"2024-04-25T01:21:51.472438+0000","last_became_active":"2024-04-25T01:21:46.104163+0000","last_became_peered":"2024-04-25T01:21:46.104163+0000","last_unstale":"2024-04-25T01:21:51.472438+0000","last_undegraded":"2024-04-25T01:21:51.472438+0000","last_fullsized":"2024-04-25T01:21:51.472438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:27.583834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:55.386967+0000","last_change":"2024-04-25T01:21:45.076242+0000","last_active":"2024-04-25T01:21:55.386967+0000","last_peered":"2024-04-25T01:21:55.386967+0000","last_clean":"2024-04-25T01:21:55.386967+0000","last_became_active":"2024-04-25T01:21:45.075952+0000","last_became_peered":"2024-04-25T01:21:45.075952+0000","last_unstale":"2024-04-25T01:21:55.386967+0000","last_undegraded":"2024-04-25T01:21:55.386967+0000","last_fullsized":"2024-04-25T01:21:55.386967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:52.481389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473757+0000","last_change":"2024-04-25T01:21:50.128837+0000","last_active":"2024-04-25T01:21:51.473757+0000","last_peered":"2024-04-25T01:21:51.473757+0000","last_clean":"2024-04-25T01:21:51.473757+0000","last_became_active":"2024-04-25T01:21:50.128462+0000","last_became_peered":"2024-04-25T01:21:50.128462+0000","last_unstale":"2024-04-25T01:21:51.473757+0000","last_undegraded":"2024-04-25T01:21:51.473757+0000","last_fullsized":"2024-04-25T01:21:51.473757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:42.606728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118967+0000","last_change":"2024-04-25T01:21:46.100341+0000","last_active":"2024-04-25T01:21:51.118967+0000","last_peered":"2024-04-25T01:21:51.118967+0000","last_clean":"2024-04-25T01:21:51.118967+0000","last_became_active":"2024-04-25T01:21:46.099961+0000","last_became_peered":"2024-04-25T01:21:46.099961+0000","last_unstale":"2024-04-25T01:21:51.118967+0000","last_undegraded":"2024-04-25T01:21:51.118967+0000","last_fullsized":"2024-04-25T01:21:51.118967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:15.176543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473450+0000","last_change":"2024-04-25T01:21:45.077748+0000","last_active":"2024-04-25T01:21:51.473450+0000","last_peered":"2024-04-25T01:21:51.473450+0000","last_clean":"2024-04-25T01:21:51.473450+0000","last_became_active":"2024-04-25T01:21:45.077494+0000","last_became_peered":"2024-04-25T01:21:45.077494+0000","last_unstale":"2024-04-25T01:21:51.473450+0000","last_undegraded":"2024-04-25T01:21:51.473450+0000","last_fullsized":"2024-04-25T01:21:51.473450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:25.950316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118695+0000","last_change":"2024-04-25T01:21:50.126572+0000","last_active":"2024-04-25T01:21:51.118695+0000","last_peered":"2024-04-25T01:21:51.118695+0000","last_clean":"2024-04-25T01:21:51.118695+0000","last_became_active":"2024-04-25T01:21:50.126013+0000","last_became_peered":"2024-04-25T01:21:50.126013+0000","last_unstale":"2024-04-25T01:21:51.118695+0000","last_undegraded":"2024-04-25T01:21:51.118695+0000","last_fullsized":"2024-04-25T01:21:51.118695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:19.877429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471878+0000","last_change":"2024-04-25T01:21:46.091135+0000","last_active":"2024-04-25T01:21:51.471878+0000","last_peered":"2024-04-25T01:21:51.471878+0000","last_clean":"2024-04-25T01:21:51.471878+0000","last_became_active":"2024-04-25T01:21:46.090493+0000","last_became_peered":"2024-04-25T01:21:46.090493+0000","last_unstale":"2024-04-25T01:21:51.471878+0000","last_undegraded":"2024-04-25T01:21:51.471878+0000","last_fullsized":"2024-04-25T01:21:51.471878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472529+0000","last_change":"2024-04-25T01:21:45.075179+0000","last_active":"2024-04-25T01:21:51.472529+0000","last_peered":"2024-04-25T01:21:51.472529+0000","last_clean":"2024-04-25T01:21:51.472529+0000","last_became_active":"2024-04-25T01:21:45.074903+0000","last_became_peered":"2024-04-25T01:21:45.074903+0000","last_unstale":"2024-04-25T01:21:51.472529+0000","last_undegraded":"2024-04-25T01:21:51.472529+0000","last_fullsized":"2024-04-25T01:21:51.472529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471819+0000","last_change":"2024-04-25T01:21:50.126406+0000","last_active":"2024-04-25T01:21:51.471819+0000","last_peered":"2024-04-25T01:21:51.471819+0000","last_clean":"2024-04-25T01:21:51.471819+0000","last_became_active":"2024-04-25T01:21:50.126029+0000","last_became_peered":"2024-04-25T01:21:50.126029+0000","last_unstale":"2024-04-25T01:21:51.471819+0000","last_undegraded":"2024-04-25T01:21:51.471819+0000","last_fullsized":"2024-04-25T01:21:51.471819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:38.806941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472398+0000","last_change":"2024-04-25T01:21:46.100746+0000","last_active":"2024-04-25T01:21:51.472398+0000","last_peered":"2024-04-25T01:21:51.472398+0000","last_clean":"2024-04-25T01:21:51.472398+0000","last_became_active":"2024-04-25T01:21:46.100475+0000","last_became_peered":"2024-04-25T01:21:46.100475+0000","last_unstale":"2024-04-25T01:21:51.472398+0000","last_undegraded":"2024-04-25T01:21:51.472398+0000","last_fullsized":"2024-04-25T01:21:51.472398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:27.802903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119162+0000","last_change":"2024-04-25T01:21:45.082455+0000","last_active":"2024-04-25T01:21:51.119162+0000","last_peered":"2024-04-25T01:21:51.119162+0000","last_clean":"2024-04-25T01:21:51.119162+0000","last_became_active":"2024-04-25T01:21:45.082135+0000","last_became_peered":"2024-04-25T01:21:45.082135+0000","last_unstale":"2024-04-25T01:21:51.119162+0000","last_undegraded":"2024-04-25T01:21:51.119162+0000","last_fullsized":"2024-04-25T01:21:51.119162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:06.296153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471896+0000","last_change":"2024-04-25T01:21:50.184859+0000","last_active":"2024-04-25T01:21:51.471896+0000","last_peered":"2024-04-25T01:21:51.471896+0000","last_clean":"2024-04-25T01:21:51.471896+0000","last_became_active":"2024-04-25T01:21:50.184475+0000","last_became_peered":"2024-04-25T01:21:50.184475+0000","last_unstale":"2024-04-25T01:21:51.471896+0000","last_undegraded":"2024-04-25T01:21:51.471896+0000","last_fullsized":"2024-04-25T01:21:51.471896+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:00.042952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472589+0000","last_change":"2024-04-25T01:21:50.184894+0000","last_active":"2024-04-25T01:21:51.472589+0000","last_peered":"2024-04-25T01:21:51.472589+0000","last_clean":"2024-04-25T01:21:51.472589+0000","last_became_active":"2024-04-25T01:21:50.184392+0000","last_became_peered":"2024-04-25T01:21:50.184392+0000","last_unstale":"2024-04-25T01:21:51.472589+0000","last_undegraded":"2024-04-25T01:21:51.472589+0000","last_fullsized":"2024-04-25T01:21:51.472589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:31.842249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472948+0000","last_change":"2024-04-25T01:21:46.092888+0000","last_active":"2024-04-25T01:21:51.472948+0000","last_peered":"2024-04-25T01:21:51.472948+0000","last_clean":"2024-04-25T01:21:51.472948+0000","last_became_active":"2024-04-25T01:21:46.092607+0000","last_became_peered":"2024-04-25T01:21:46.092607+0000","last_unstale":"2024-04-25T01:21:51.472948+0000","last_undegraded":"2024-04-25T01:21:51.472948+0000","last_fullsized":"2024-04-25T01:21:51.472948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:22.971333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472329+0000","last_change":"2024-04-25T01:21:45.070532+0000","last_active":"2024-04-25T01:21:51.472329+0000","last_peered":"2024-04-25T01:21:51.472329+0000","last_clean":"2024-04-25T01:21:51.472329+0000","last_became_active":"2024-04-25T01:21:45.070256+0000","last_became_peered":"2024-04-25T01:21:45.070256+0000","last_unstale":"2024-04-25T01:21:51.472329+0000","last_undegraded":"2024-04-25T01:21:51.472329+0000","last_fullsized":"2024-04-25T01:21:51.472329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119151+0000","last_change":"2024-04-25T01:21:50.130056+0000","last_active":"2024-04-25T01:21:51.119151+0000","last_peered":"2024-04-25T01:21:51.119151+0000","last_clean":"2024-04-25T01:21:51.119151+0000","last_became_active":"2024-04-25T01:21:50.129541+0000","last_became_peered":"2024-04-25T01:21:50.129541+0000","last_unstale":"2024-04-25T01:21:51.119151+0000","last_undegraded":"2024-04-25T01:21:51.119151+0000","last_fullsized":"2024-04-25T01:21:51.119151+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471871+0000","last_change":"2024-04-25T01:21:46.083728+0000","last_active":"2024-04-25T01:21:51.471871+0000","last_peered":"2024-04-25T01:21:51.471871+0000","last_clean":"2024-04-25T01:21:51.471871+0000","last_became_active":"2024-04-25T01:21:46.083458+0000","last_became_peered":"2024-04-25T01:21:46.083458+0000","last_unstale":"2024-04-25T01:21:51.471871+0000","last_undegraded":"2024-04-25T01:21:51.471871+0000","last_fullsized":"2024-04-25T01:21:51.471871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472093+0000","last_change":"2024-04-25T01:21:45.077308+0000","last_active":"2024-04-25T01:21:51.472093+0000","last_peered":"2024-04-25T01:21:51.472093+0000","last_clean":"2024-04-25T01:21:51.472093+0000","last_became_active":"2024-04-25T01:21:45.076815+0000","last_became_peered":"2024-04-25T01:21:45.076815+0000","last_unstale":"2024-04-25T01:21:51.472093+0000","last_undegraded":"2024-04-25T01:21:51.472093+0000","last_fullsized":"2024-04-25T01:21:51.472093+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:43.833096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118432+0000","last_change":"2024-04-25T01:21:45.079268+0000","last_active":"2024-04-25T01:21:51.118432+0000","last_peered":"2024-04-25T01:21:51.118432+0000","last_clean":"2024-04-25T01:21:51.118432+0000","last_became_active":"2024-04-25T01:21:45.078934+0000","last_became_peered":"2024-04-25T01:21:45.078934+0000","last_unstale":"2024-04-25T01:21:51.118432+0000","last_undegraded":"2024-04-25T01:21:51.118432+0000","last_fullsized":"2024-04-25T01:21:51.118432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472316+0000","last_change":"2024-04-25T01:21:46.086329+0000","last_active":"2024-04-25T01:21:51.472316+0000","last_peered":"2024-04-25T01:21:51.472316+0000","last_clean":"2024-04-25T01:21:51.472316+0000","last_became_active":"2024-04-25T01:21:46.086069+0000","last_became_peered":"2024-04-25T01:21:46.086069+0000","last_unstale":"2024-04-25T01:21:51.472316+0000","last_undegraded":"2024-04-25T01:21:51.472316+0000","last_fullsized":"2024-04-25T01:21:51.472316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119441+0000","last_change":"2024-04-25T01:21:50.122334+0000","last_active":"2024-04-25T01:21:51.119441+0000","last_peered":"2024-04-25T01:21:51.119441+0000","last_clean":"2024-04-25T01:21:51.119441+0000","last_became_active":"2024-04-25T01:21:50.121974+0000","last_became_peered":"2024-04-25T01:21:50.121974+0000","last_unstale":"2024-04-25T01:21:51.119441+0000","last_undegraded":"2024-04-25T01:21:51.119441+0000","last_fullsized":"2024-04-25T01:21:51.119441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:59.115674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473488+0000","last_change":"2024-04-25T01:21:45.082740+0000","last_active":"2024-04-25T01:21:51.473488+0000","last_peered":"2024-04-25T01:21:51.473488+0000","last_clean":"2024-04-25T01:21:51.473488+0000","last_became_active":"2024-04-25T01:21:45.082439+0000","last_became_peered":"2024-04-25T01:21:45.082439+0000","last_unstale":"2024-04-25T01:21:51.473488+0000","last_undegraded":"2024-04-25T01:21:51.473488+0000","last_fullsized":"2024-04-25T01:21:51.473488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:52.481389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118237+0000","last_change":"2024-04-25T01:21:46.088483+0000","last_active":"2024-04-25T01:21:51.118237+0000","last_peered":"2024-04-25T01:21:51.118237+0000","last_clean":"2024-04-25T01:21:51.118237+0000","last_became_active":"2024-04-25T01:21:46.088113+0000","last_became_peered":"2024-04-25T01:21:46.088113+0000","last_unstale":"2024-04-25T01:21:51.118237+0000","last_undegraded":"2024-04-25T01:21:51.118237+0000","last_fullsized":"2024-04-25T01:21:51.118237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:22.055684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472316+0000","last_change":"2024-04-25T01:21:50.125319+0000","last_active":"2024-04-25T01:21:51.472316+0000","last_peered":"2024-04-25T01:21:51.472316+0000","last_clean":"2024-04-25T01:21:51.472316+0000","last_became_active":"2024-04-25T01:21:50.124926+0000","last_became_peered":"2024-04-25T01:21:50.124926+0000","last_unstale":"2024-04-25T01:21:51.472316+0000","last_undegraded":"2024-04-25T01:21:51.472316+0000","last_fullsized":"2024-04-25T01:21:51.472316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:20.747339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118296+0000","last_change":"2024-04-25T01:21:46.093690+0000","last_active":"2024-04-25T01:21:51.118296+0000","last_peered":"2024-04-25T01:21:51.118296+0000","last_clean":"2024-04-25T01:21:51.118296+0000","last_became_active":"2024-04-25T01:21:46.093392+0000","last_became_peered":"2024-04-25T01:21:46.093392+0000","last_unstale":"2024-04-25T01:21:51.118296+0000","last_undegraded":"2024-04-25T01:21:51.118296+0000","last_fullsized":"2024-04-25T01:21:51.118296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119144+0000","last_change":"2024-04-25T01:21:46.089771+0000","last_active":"2024-04-25T01:21:51.119144+0000","last_peered":"2024-04-25T01:21:51.119144+0000","last_clean":"2024-04-25T01:21:51.119144+0000","last_became_active":"2024-04-25T01:21:46.089489+0000","last_became_peered":"2024-04-25T01:21:46.089489+0000","last_unstale":"2024-04-25T01:21:51.119144+0000","last_undegraded":"2024-04-25T01:21:51.119144+0000","last_fullsized":"2024-04-25T01:21:51.119144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:16.213667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473158+0000","last_change":"2024-04-25T01:21:46.099812+0000","last_active":"2024-04-25T01:21:51.473158+0000","last_peered":"2024-04-25T01:21:51.473158+0000","last_clean":"2024-04-25T01:21:51.473158+0000","last_became_active":"2024-04-25T01:21:46.099508+0000","last_became_peered":"2024-04-25T01:21:46.099508+0000","last_unstale":"2024-04-25T01:21:51.473158+0000","last_undegraded":"2024-04-25T01:21:51.473158+0000","last_fullsized":"2024-04-25T01:21:51.473158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:45.024693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472233+0000","last_change":"2024-04-25T01:21:46.083637+0000","last_active":"2024-04-25T01:21:51.472233+0000","last_peered":"2024-04-25T01:21:51.472233+0000","last_clean":"2024-04-25T01:21:51.472233+0000","last_became_active":"2024-04-25T01:21:46.083371+0000","last_became_peered":"2024-04-25T01:21:46.083371+0000","last_unstale":"2024-04-25T01:21:51.472233+0000","last_undegraded":"2024-04-25T01:21:51.472233+0000","last_fullsized":"2024-04-25T01:21:51.472233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473043+0000","last_change":"2024-04-25T01:21:46.087822+0000","last_active":"2024-04-25T01:21:51.473043+0000","last_peered":"2024-04-25T01:21:51.473043+0000","last_clean":"2024-04-25T01:21:51.473043+0000","last_became_active":"2024-04-25T01:21:46.087555+0000","last_became_peered":"2024-04-25T01:21:46.087555+0000","last_unstale":"2024-04-25T01:21:51.473043+0000","last_undegraded":"2024-04-25T01:21:51.473043+0000","last_fullsized":"2024-04-25T01:21:51.473043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473261+0000","last_change":"2024-04-25T01:21:46.092978+0000","last_active":"2024-04-25T01:21:51.473261+0000","last_peered":"2024-04-25T01:21:51.473261+0000","last_clean":"2024-04-25T01:21:51.473261+0000","last_became_active":"2024-04-25T01:21:46.092720+0000","last_became_peered":"2024-04-25T01:21:46.092720+0000","last_unstale":"2024-04-25T01:21:51.473261+0000","last_undegraded":"2024-04-25T01:21:51.473261+0000","last_fullsized":"2024-04-25T01:21:51.473261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119314+0000","last_change":"2024-04-25T01:21:46.100969+0000","last_active":"2024-04-25T01:21:51.119314+0000","last_peered":"2024-04-25T01:21:51.119314+0000","last_clean":"2024-04-25T01:21:51.119314+0000","last_became_active":"2024-04-25T01:21:46.100687+0000","last_became_peered":"2024-04-25T01:21:46.100687+0000","last_unstale":"2024-04-25T01:21:51.119314+0000","last_undegraded":"2024-04-25T01:21:51.119314+0000","last_fullsized":"2024-04-25T01:21:51.119314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472539+0000","last_change":"2024-04-25T01:21:46.087688+0000","last_active":"2024-04-25T01:21:51.472539+0000","last_peered":"2024-04-25T01:21:51.472539+0000","last_clean":"2024-04-25T01:21:51.472539+0000","last_became_active":"2024-04-25T01:21:46.087422+0000","last_became_peered":"2024-04-25T01:21:46.087422+0000","last_unstale":"2024-04-25T01:21:51.472539+0000","last_undegraded":"2024-04-25T01:21:51.472539+0000","last_fullsized":"2024-04-25T01:21:51.472539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472217+0000","last_change":"2024-04-25T01:21:46.096585+0000","last_active":"2024-04-25T01:21:51.472217+0000","last_peered":"2024-04-25T01:21:51.472217+0000","last_clean":"2024-04-25T01:21:51.472217+0000","last_became_active":"2024-04-25T01:21:46.096299+0000","last_became_peered":"2024-04-25T01:21:46.096299+0000","last_unstale":"2024-04-25T01:21:51.472217+0000","last_undegraded":"2024-04-25T01:21:51.472217+0000","last_fullsized":"2024-04-25T01:21:51.472217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472814+0000","last_change":"2024-04-25T01:21:46.085048+0000","last_active":"2024-04-25T01:21:51.472814+0000","last_peered":"2024-04-25T01:21:51.472814+0000","last_clean":"2024-04-25T01:21:51.472814+0000","last_became_active":"2024-04-25T01:21:46.084766+0000","last_became_peered":"2024-04-25T01:21:46.084766+0000","last_unstale":"2024-04-25T01:21:51.472814+0000","last_undegraded":"2024-04-25T01:21:51.472814+0000","last_fullsized":"2024-04-25T01:21:51.472814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473111+0000","last_change":"2024-04-25T01:21:46.105035+0000","last_active":"2024-04-25T01:21:51.473111+0000","last_peered":"2024-04-25T01:21:51.473111+0000","last_clean":"2024-04-25T01:21:51.473111+0000","last_became_active":"2024-04-25T01:21:46.104776+0000","last_became_peered":"2024-04-25T01:21:46.104776+0000","last_unstale":"2024-04-25T01:21:51.473111+0000","last_undegraded":"2024-04-25T01:21:51.473111+0000","last_fullsized":"2024-04-25T01:21:51.473111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471992+0000","last_change":"2024-04-25T01:21:46.100414+0000","last_active":"2024-04-25T01:21:51.471992+0000","last_peered":"2024-04-25T01:21:51.471992+0000","last_clean":"2024-04-25T01:21:51.471992+0000","last_became_active":"2024-04-25T01:21:46.100140+0000","last_became_peered":"2024-04-25T01:21:46.100140+0000","last_unstale":"2024-04-25T01:21:51.471992+0000","last_undegraded":"2024-04-25T01:21:51.471992+0000","last_fullsized":"2024-04-25T01:21:51.471992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:55.076328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118784+0000","last_change":"2024-04-25T01:21:46.092761+0000","last_active":"2024-04-25T01:21:51.118784+0000","last_peered":"2024-04-25T01:21:51.118784+0000","last_clean":"2024-04-25T01:21:51.118784+0000","last_became_active":"2024-04-25T01:21:46.092230+0000","last_became_peered":"2024-04-25T01:21:46.092230+0000","last_unstale":"2024-04-25T01:21:51.118784+0000","last_undegraded":"2024-04-25T01:21:51.118784+0000","last_fullsized":"2024-04-25T01:21:51.118784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472724+0000","last_change":"2024-04-25T01:21:46.104951+0000","last_active":"2024-04-25T01:21:51.472724+0000","last_peered":"2024-04-25T01:21:51.472724+0000","last_clean":"2024-04-25T01:21:51.472724+0000","last_became_active":"2024-04-25T01:21:46.104662+0000","last_became_peered":"2024-04-25T01:21:46.104662+0000","last_unstale":"2024-04-25T01:21:51.472724+0000","last_undegraded":"2024-04-25T01:21:51.472724+0000","last_fullsized":"2024-04-25T01:21:51.472724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:46.588303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119527+0000","last_change":"2024-04-25T01:21:46.091860+0000","last_active":"2024-04-25T01:21:51.119527+0000","last_peered":"2024-04-25T01:21:51.119527+0000","last_clean":"2024-04-25T01:21:51.119527+0000","last_became_active":"2024-04-25T01:21:46.091569+0000","last_became_peered":"2024-04-25T01:21:46.091569+0000","last_unstale":"2024-04-25T01:21:51.119527+0000","last_undegraded":"2024-04-25T01:21:51.119527+0000","last_fullsized":"2024-04-25T01:21:51.119527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:57.449234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473437+0000","last_change":"2024-04-25T01:21:46.099956+0000","last_active":"2024-04-25T01:21:51.473437+0000","last_peered":"2024-04-25T01:21:51.473437+0000","last_clean":"2024-04-25T01:21:51.473437+0000","last_became_active":"2024-04-25T01:21:46.099687+0000","last_became_peered":"2024-04-25T01:21:46.099687+0000","last_unstale":"2024-04-25T01:21:51.473437+0000","last_undegraded":"2024-04-25T01:21:51.473437+0000","last_fullsized":"2024-04-25T01:21:51.473437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:30.138160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472108+0000","last_change":"2024-04-25T01:21:46.102084+0000","last_active":"2024-04-25T01:21:51.472108+0000","last_peered":"2024-04-25T01:21:51.472108+0000","last_clean":"2024-04-25T01:21:51.472108+0000","last_became_active":"2024-04-25T01:21:46.101539+0000","last_became_peered":"2024-04-25T01:21:46.101539+0000","last_unstale":"2024-04-25T01:21:51.472108+0000","last_undegraded":"2024-04-25T01:21:51.472108+0000","last_fullsized":"2024-04-25T01:21:51.472108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473063+0000","last_change":"2024-04-25T01:21:46.089149+0000","last_active":"2024-04-25T01:21:51.473063+0000","last_peered":"2024-04-25T01:21:51.473063+0000","last_clean":"2024-04-25T01:21:51.473063+0000","last_became_active":"2024-04-25T01:21:46.088845+0000","last_became_peered":"2024-04-25T01:21:46.088845+0000","last_unstale":"2024-04-25T01:21:51.473063+0000","last_undegraded":"2024-04-25T01:21:51.473063+0000","last_fullsized":"2024-04-25T01:21:51.473063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472391+0000","last_change":"2024-04-25T01:21:46.087764+0000","last_active":"2024-04-25T01:21:51.472391+0000","last_peered":"2024-04-25T01:21:51.472391+0000","last_clean":"2024-04-25T01:21:51.472391+0000","last_became_active":"2024-04-25T01:21:46.087481+0000","last_became_peered":"2024-04-25T01:21:46.087481+0000","last_unstale":"2024-04-25T01:21:51.472391+0000","last_undegraded":"2024-04-25T01:21:51.472391+0000","last_fullsized":"2024-04-25T01:21:51.472391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:16.213667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473322+0000","last_change":"2024-04-25T01:21:46.093981+0000","last_active":"2024-04-25T01:21:51.473322+0000","last_peered":"2024-04-25T01:21:51.473322+0000","last_clean":"2024-04-25T01:21:51.473322+0000","last_became_active":"2024-04-25T01:21:46.093729+0000","last_became_peered":"2024-04-25T01:21:46.093729+0000","last_unstale":"2024-04-25T01:21:51.473322+0000","last_undegraded":"2024-04-25T01:21:51.473322+0000","last_fullsized":"2024-04-25T01:21:51.473322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118866+0000","last_change":"2024-04-25T01:21:46.094258+0000","last_active":"2024-04-25T01:21:51.118866+0000","last_peered":"2024-04-25T01:21:51.118866+0000","last_clean":"2024-04-25T01:21:51.118866+0000","last_became_active":"2024-04-25T01:21:46.093633+0000","last_became_peered":"2024-04-25T01:21:46.093633+0000","last_unstale":"2024-04-25T01:21:51.118866+0000","last_undegraded":"2024-04-25T01:21:51.118866+0000","last_fullsized":"2024-04-25T01:21:51.118866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:52.240635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118920+0000","last_change":"2024-04-25T01:21:46.093764+0000","last_active":"2024-04-25T01:21:51.118920+0000","last_peered":"2024-04-25T01:21:51.118920+0000","last_clean":"2024-04-25T01:21:51.118920+0000","last_became_active":"2024-04-25T01:21:46.093468+0000","last_became_peered":"2024-04-25T01:21:46.093468+0000","last_unstale":"2024-04-25T01:21:51.118920+0000","last_undegraded":"2024-04-25T01:21:51.118920+0000","last_fullsized":"2024-04-25T01:21:51.118920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119290+0000","last_change":"2024-04-25T01:21:46.096039+0000","last_active":"2024-04-25T01:21:51.119290+0000","last_peered":"2024-04-25T01:21:51.119290+0000","last_clean":"2024-04-25T01:21:51.119290+0000","last_became_active":"2024-04-25T01:21:46.095773+0000","last_became_peered":"2024-04-25T01:21:46.095773+0000","last_unstale":"2024-04-25T01:21:51.119290+0000","last_undegraded":"2024-04-25T01:21:51.119290+0000","last_fullsized":"2024-04-25T01:21:51.119290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472805+0000","last_change":"2024-04-25T01:21:46.106320+0000","last_active":"2024-04-25T01:21:51.472805+0000","last_peered":"2024-04-25T01:21:51.472805+0000","last_clean":"2024-04-25T01:21:51.472805+0000","last_became_active":"2024-04-25T01:21:46.106064+0000","last_became_peered":"2024-04-25T01:21:46.106064+0000","last_unstale":"2024-04-25T01:21:51.472805+0000","last_undegraded":"2024-04-25T01:21:51.472805+0000","last_fullsized":"2024-04-25T01:21:51.472805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:24.344315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471632+0000","last_change":"2024-04-25T01:21:46.093521+0000","last_active":"2024-04-25T01:21:51.471632+0000","last_peered":"2024-04-25T01:21:51.471632+0000","last_clean":"2024-04-25T01:21:51.471632+0000","last_became_active":"2024-04-25T01:21:46.093269+0000","last_became_peered":"2024-04-25T01:21:46.093269+0000","last_unstale":"2024-04-25T01:21:51.471632+0000","last_undegraded":"2024-04-25T01:21:51.471632+0000","last_fullsized":"2024-04-25T01:21:51.471632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472027+0000","last_change":"2024-04-25T01:21:46.102728+0000","last_active":"2024-04-25T01:21:51.472027+0000","last_peered":"2024-04-25T01:21:51.472027+0000","last_clean":"2024-04-25T01:21:51.472027+0000","last_became_active":"2024-04-25T01:21:46.102475+0000","last_became_peered":"2024-04-25T01:21:46.102475+0000","last_unstale":"2024-04-25T01:21:51.472027+0000","last_undegraded":"2024-04-25T01:21:51.472027+0000","last_fullsized":"2024-04-25T01:21:51.472027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472990+0000","last_change":"2024-04-25T01:21:46.101690+0000","last_active":"2024-04-25T01:21:51.472990+0000","last_peered":"2024-04-25T01:21:51.472990+0000","last_clean":"2024-04-25T01:21:51.472990+0000","last_became_active":"2024-04-25T01:21:46.101404+0000","last_became_peered":"2024-04-25T01:21:46.101404+0000","last_unstale":"2024-04-25T01:21:51.472990+0000","last_undegraded":"2024-04-25T01:21:51.472990+0000","last_fullsized":"2024-04-25T01:21:51.472990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472132+0000","last_change":"2024-04-25T01:21:46.103900+0000","last_active":"2024-04-25T01:21:51.472132+0000","last_peered":"2024-04-25T01:21:51.472132+0000","last_clean":"2024-04-25T01:21:51.472132+0000","last_became_active":"2024-04-25T01:21:46.103633+0000","last_became_peered":"2024-04-25T01:21:46.103633+0000","last_unstale":"2024-04-25T01:21:51.472132+0000","last_undegraded":"2024-04-25T01:21:51.472132+0000","last_fullsized":"2024-04-25T01:21:51.472132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:42.961329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472786+0000","last_change":"2024-04-25T01:21:46.102312+0000","last_active":"2024-04-25T01:21:51.472786+0000","last_peered":"2024-04-25T01:21:51.472786+0000","last_clean":"2024-04-25T01:21:51.472786+0000","last_became_active":"2024-04-25T01:21:46.101113+0000","last_became_peered":"2024-04-25T01:21:46.101113+0000","last_unstale":"2024-04-25T01:21:51.472786+0000","last_undegraded":"2024-04-25T01:21:51.472786+0000","last_fullsized":"2024-04-25T01:21:51.472786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:46.038059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473192+0000","last_change":"2024-04-25T01:21:46.098593+0000","last_active":"2024-04-25T01:21:51.473192+0000","last_peered":"2024-04-25T01:21:51.473192+0000","last_clean":"2024-04-25T01:21:51.473192+0000","last_became_active":"2024-04-25T01:21:46.098289+0000","last_became_peered":"2024-04-25T01:21:46.098289+0000","last_unstale":"2024-04-25T01:21:51.473192+0000","last_undegraded":"2024-04-25T01:21:51.473192+0000","last_fullsized":"2024-04-25T01:21:51.473192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473504+0000","last_change":"2024-04-25T01:21:46.093954+0000","last_active":"2024-04-25T01:21:51.473504+0000","last_peered":"2024-04-25T01:21:51.473504+0000","last_clean":"2024-04-25T01:21:51.473504+0000","last_became_active":"2024-04-25T01:21:46.093658+0000","last_became_peered":"2024-04-25T01:21:46.093658+0000","last_unstale":"2024-04-25T01:21:51.473504+0000","last_undegraded":"2024-04-25T01:21:51.473504+0000","last_fullsized":"2024-04-25T01:21:51.473504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472718+0000","last_change":"2024-04-25T01:21:46.108321+0000","last_active":"2024-04-25T01:21:51.472718+0000","last_peered":"2024-04-25T01:21:51.472718+0000","last_clean":"2024-04-25T01:21:51.472718+0000","last_became_active":"2024-04-25T01:21:46.108056+0000","last_became_peered":"2024-04-25T01:21:46.108056+0000","last_unstale":"2024-04-25T01:21:51.472718+0000","last_undegraded":"2024-04-25T01:21:51.472718+0000","last_fullsized":"2024-04-25T01:21:51.472718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:22:08.502 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T01:22:08.502 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T01:22:08.503 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T01:22:08.503 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T01:22:08.503 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T01:22:08.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.653+0000 7f695ad3b640 1 Processor -- start 2024-04-25T01:22:08.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.653+0000 7f695ad3b640 1 -- start start 2024-04-25T01:22:08.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.654+0000 7f695ad3b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f695406ed60 0x7f695415f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.654+0000 7f695ad3b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6954070e60 con 0x7f6954143d20 2024-04-25T01:22:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.654+0000 7f695ad3b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6954071140 con 0x7f6954143940 2024-04-25T01:22:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.654+0000 7f695ad3b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6954071420 con 0x7f695406ed60 2024-04-25T01:22:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.654+0000 7f6953fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6954143d20 0x7f695415ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56396/0 (socket says 172.21.15.77:56396) 2024-04-25T01:22:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.654+0000 7f6953fff640 1 -- 172.21.15.77:0/1823497925 learned_addr learned my addr 172.21.15.77:0/1823497925 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.654+0000 7f69592d8640 1 --2- 172.21.15.77:0/1823497925 >> v2:172.21.15.121:3300/0 conn(0x7f695406ed60 0x7f695415f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.654+0000 7f69537fe640 1 -- 172.21.15.77:0/1823497925 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2443287105 0 0) 0x7f6954070e60 con 0x7f6954143d20 2024-04-25T01:22:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.654+0000 7f69537fe640 1 -- 172.21.15.77:0/1823497925 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f693c003520 con 0x7f6954143d20 2024-04-25T01:22:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.655+0000 7f69592d8640 1 -- 172.21.15.77:0/1823497925 >> v1:172.21.15.121:6789/0 conn(0x7f6954143940 legacy=0x7f695406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.655+0000 7f69537fe640 1 -- 172.21.15.77:0/1823497925 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2213732124 0 0) 0x7f6954071140 con 0x7f6954143940 2024-04-25T01:22:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.655+0000 7f69592d8640 1 -- 172.21.15.77:0/1823497925 >> v1:172.21.15.77:6789/0 conn(0x7f6954143d20 legacy=0x7f695415ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.655+0000 7f69592d8640 1 -- 172.21.15.77:0/1823497925 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6954071700 con 0x7f695406ed60 2024-04-25T01:22:08.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.655+0000 7f69592d8640 1 --2- 172.21.15.77:0/1823497925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f695406ed60 0x7f695415f400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f694c009dc0 tx=0x7f694c02a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bd61ebfdda2eac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:08.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.655+0000 7f69537fe640 1 -- 172.21.15.77:0/1823497925 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694c007740 con 0x7f695406ed60 2024-04-25T01:22:08.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.656+0000 7f69537fe640 1 -- 172.21.15.77:0/1823497925 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f694c0078e0 con 0x7f695406ed60 2024-04-25T01:22:08.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.656+0000 7f69537fe640 1 -- 172.21.15.77:0/1823497925 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694c007c90 con 0x7f695406ed60 2024-04-25T01:22:08.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.656+0000 7f695ad3b640 1 -- 172.21.15.77:0/1823497925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f695406ed60 msgr2=0x7f695415f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.656+0000 7f695ad3b640 1 --2- 172.21.15.77:0/1823497925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f695406ed60 0x7f695415f400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f694c009dc0 tx=0x7f694c02a730 comp rx=0 tx=0).stop 2024-04-25T01:22:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.656+0000 7f695ad3b640 1 -- 172.21.15.77:0/1823497925 shutdown_connections 2024-04-25T01:22:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.656+0000 7f695ad3b640 1 --2- 172.21.15.77:0/1823497925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f695406ed60 0x7f695415f400 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.656+0000 7f695ad3b640 1 -- 172.21.15.77:0/1823497925 >> 172.21.15.77:0/1823497925 conn(0x7f69540b1820 msgr2=0x7f69540b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.656+0000 7f695ad3b640 1 -- 172.21.15.77:0/1823497925 shutdown_connections 2024-04-25T01:22:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.656+0000 7f695ad3b640 1 -- 172.21.15.77:0/1823497925 wait complete. 2024-04-25T01:22:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.657+0000 7f695ad3b640 1 Processor -- start 2024-04-25T01:22:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.657+0000 7f695ad3b640 1 -- start start 2024-04-25T01:22:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f695ad3b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695406ed60 0x7f6954140560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f695ad3b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6954143940 0x7f6954142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f695ad3b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6954143d20 0x7f69540cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f695ad3b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f695406d540 con 0x7f695406ed60 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f695ad3b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6954072ec0 con 0x7f6954143d20 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f695ad3b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f695406d6c0 con 0x7f6954143940 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f6958ad7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695406ed60 0x7f6954140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f6958ad7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695406ed60 0x7f6954140560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56036/0 (socket says 172.21.15.77:56036) 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f6958ad7640 1 -- 172.21.15.77:0/2928250559 learned_addr learned my addr 172.21.15.77:0/2928250559 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f6953fff640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6954143940 0x7f6954142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f69592d8640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6954143d20 0x7f69540cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f6958ad7640 1 -- 172.21.15.77:0/2928250559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6954143940 msgr2=0x7f6954142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f6958ad7640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6954143940 0x7f6954142710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f6958ad7640 1 -- 172.21.15.77:0/2928250559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6954143d20 msgr2=0x7f69540cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f6958ad7640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6954143d20 0x7f69540cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f6958ad7640 1 -- 172.21.15.77:0/2928250559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69541ef480 con 0x7f695406ed60 2024-04-25T01:22:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.658+0000 7f6958ad7640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695406ed60 0x7f6954140560 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f694400c4b0 tx=0x7f694400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.659+0000 7f69517fa640 1 -- 172.21.15.77:0/2928250559 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6944017070 con 0x7f695406ed60 2024-04-25T01:22:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.659+0000 7f69517fa640 1 -- 172.21.15.77:0/2928250559 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6944005480 con 0x7f695406ed60 2024-04-25T01:22:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.659+0000 7f695ad3b640 1 -- 172.21.15.77:0/2928250559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69541ef760 con 0x7f695406ed60 2024-04-25T01:22:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.659+0000 7f69517fa640 1 -- 172.21.15.77:0/2928250559 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6944003830 con 0x7f695406ed60 2024-04-25T01:22:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.659+0000 7f695ad3b640 1 -- 172.21.15.77:0/2928250559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69541effa0 con 0x7f695406ed60 2024-04-25T01:22:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.660+0000 7f695ad3b640 1 -- 172.21.15.77:0/2928250559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6954068fb0 con 0x7f695406ed60 2024-04-25T01:22:08.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.660+0000 7f69517fa640 1 -- 172.21.15.77:0/2928250559 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f694401a070 con 0x7f695406ed60 2024-04-25T01:22:08.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.661+0000 7f69517fa640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f694006be30 0x7f694006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.661+0000 7f6953fff640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f694006be30 0x7f694006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.661+0000 7f69517fa640 1 -- 172.21.15.77:0/2928250559 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f694408a620 con 0x7f695406ed60 2024-04-25T01:22:08.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.661+0000 7f6953fff640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f694006be30 0x7f694006e2f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f69541f05f0 tx=0x7f6948009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:08.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.665+0000 7f69517fa640 1 -- 172.21.15.77:0/2928250559 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f69541ef760 con 0x7f695406ed60 2024-04-25T01:22:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.889+0000 7f695ad3b640 1 -- 172.21.15.77:0/2928250559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f695406d150 con 0x7f695406ed60 2024-04-25T01:22:08.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.891+0000 7f69517fa640 1 -- 172.21.15.77:0/2928250559 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172934 (secure 0 0 0) 0x7f6944077070 con 0x7f695406ed60 2024-04-25T01:22:08.875 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 -- 172.21.15.77:0/2928250559 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f694006be30 msgr2=0x7f694006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f694006be30 0x7f694006e2f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f69541f05f0 tx=0x7f6948009040 comp rx=0 tx=0).stop 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 -- 172.21.15.77:0/2928250559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695406ed60 msgr2=0x7f6954140560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695406ed60 0x7f6954140560 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f694400c4b0 tx=0x7f694400c970 comp rx=0 tx=0).stop 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 -- 172.21.15.77:0/2928250559 shutdown_connections 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f694006be30 0x7f694006e2f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6954143d20 0x7f69540cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6954143940 0x7f6954142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 --2- 172.21.15.77:0/2928250559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695406ed60 0x7f6954140560 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 -- 172.21.15.77:0/2928250559 >> 172.21.15.77:0/2928250559 conn(0x7f69540b1820 msgr2=0x7f69540d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 -- 172.21.15.77:0/2928250559 shutdown_connections 2024-04-25T01:22:08.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:08.894+0000 7f695ad3b640 1 -- 172.21.15.77:0/2928250559 wait complete. 2024-04-25T01:22:08.887 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":436162236},{"type":"v1","addr":"172.21.15.77:6833","nonce":436162236}]},"active_addr":"172.21.15.77:6833/436162236","active_change":"2024-04-25T01:21:25.147209+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4102,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3470999693}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":240536759}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3618887086}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":229601152}]}]} 2024-04-25T01:22:08.890 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T01:22:08.890 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T01:22:08.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:08.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.005+0000 7f2fb5476640 1 Processor -- start 2024-04-25T01:22:08.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.005+0000 7f2fb5476640 1 -- start start 2024-04-25T01:22:08.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2fb5476640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2fb015e4c0 0x7f2fb00d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2fb5476640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2fb00d4a10 0x7f2fb00b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2fb5476640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb00d1e90 con 0x7f2fb00d2d40 2024-04-25T01:22:08.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2fb5476640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb00b54a0 con 0x7f2fb00d4a10 2024-04-25T01:22:08.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2fb5476640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb00d2130 con 0x7f2fb015e4c0 2024-04-25T01:22:08.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2fae7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2fb00d4a10 0x7f2fb00b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2faf7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2fb00d2d40 0x7f2fb00d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56406/0 (socket says 172.21.15.77:56406) 2024-04-25T01:22:08.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2faf7fe640 1 -- 172.21.15.77:0/1011698532 learned_addr learned my addr 172.21.15.77:0/1011698532 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:08.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2faeffd640 1 --2- 172.21.15.77:0/1011698532 >> v2:172.21.15.121:3300/0 conn(0x7f2fb015e4c0 0x7f2fb00d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2fae7fc640 1 -- 172.21.15.77:0/1011698532 >> v2:172.21.15.121:3300/0 conn(0x7f2fb015e4c0 msgr2=0x7f2fb00d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:08.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2fae7fc640 1 --2- 172.21.15.77:0/1011698532 >> v2:172.21.15.121:3300/0 conn(0x7f2fb015e4c0 0x7f2fb00d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2fae7fc640 1 -- 172.21.15.77:0/1011698532 >> v1:172.21.15.77:6789/0 conn(0x7f2fb00d2d40 legacy=0x7f2fb00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2fae7fc640 1 -- 172.21.15.77:0/1011698532 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb00b5780 con 0x7f2fb00d4a10 2024-04-25T01:22:08.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.006+0000 7f2fadffb640 1 -- 172.21.15.77:0/1011698532 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3560728127 0 0) 0x7f2fb00d1e90 con 0x7f2fb00d2d40 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.007+0000 7f2fae7fc640 1 --2- 172.21.15.77:0/1011698532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb00d4a10 0x7f2fb00b3ee0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0067960 tx=0x7f2fa402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50af5067090f4de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.007+0000 7f2fadffb640 1 -- 172.21.15.77:0/1011698532 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4037070 con 0x7f2fb00d4a10 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.007+0000 7f2fadffb640 1 -- 172.21.15.77:0/1011698532 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa4032440 con 0x7f2fb00d4a10 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.007+0000 7f2fadffb640 1 -- 172.21.15.77:0/1011698532 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa402ae60 con 0x7f2fb00d4a10 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.007+0000 7f2fb5476640 1 -- 172.21.15.77:0/1011698532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb00d4a10 msgr2=0x7f2fb00b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.007+0000 7f2fb5476640 1 --2- 172.21.15.77:0/1011698532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb00d4a10 0x7f2fb00b3ee0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0067960 tx=0x7f2fa402a110 comp rx=0 tx=0).stop 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.008+0000 7f2fb5476640 1 -- 172.21.15.77:0/1011698532 shutdown_connections 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.008+0000 7f2fb5476640 1 --2- 172.21.15.77:0/1011698532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb00d4a10 0x7f2fb00b3ee0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.008+0000 7f2fb5476640 1 --2- 172.21.15.77:0/1011698532 >> v2:172.21.15.121:3300/0 conn(0x7f2fb015e4c0 0x7f2fb00d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.008+0000 7f2fb5476640 1 -- 172.21.15.77:0/1011698532 >> 172.21.15.77:0/1011698532 conn(0x7f2fb00d3ef0 msgr2=0x7f2fb00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.008+0000 7f2fb5476640 1 -- 172.21.15.77:0/1011698532 shutdown_connections 2024-04-25T01:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.008+0000 7f2fb5476640 1 -- 172.21.15.77:0/1011698532 wait complete. 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.009+0000 7f2fb5476640 1 Processor -- start 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.009+0000 7f2fb5476640 1 -- start start 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.009+0000 7f2fb5476640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00d2d40 0x7f2fb00cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.009+0000 7f2fb5476640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb00d4a10 0x7f2fb0153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2fb5476640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb015e4c0 0x7f2fb015cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2fb5476640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb015ea20 con 0x7f2fb015e4c0 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2fb5476640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb015e8a0 con 0x7f2fb00d2d40 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2fb5476640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb00d1010 con 0x7f2fb00d4a10 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2faf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb015e4c0 0x7f2fb015cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2faf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb015e4c0 0x7f2fb015cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56060/0 (socket says 172.21.15.77:56060) 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2faf7fe640 1 -- 172.21.15.77:0/1985227502 learned_addr learned my addr 172.21.15.77:0/1985227502 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:08.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2faeffd640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00d2d40 0x7f2fb00cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2fae7fc640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb00d4a10 0x7f2fb0153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2faf7fe640 1 -- 172.21.15.77:0/1985227502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb00d4a10 msgr2=0x7f2fb0153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2faf7fe640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb00d4a10 0x7f2fb0153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2faf7fe640 1 -- 172.21.15.77:0/1985227502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00d2d40 msgr2=0x7f2fb00cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2faf7fe640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00d2d40 0x7f2fb00cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2faf7fe640 1 -- 172.21.15.77:0/1985227502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb00ccf60 con 0x7f2fb015e4c0 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2faf7fe640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb015e4c0 0x7f2fb015cd30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800c480 tx=0x7f2fa800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2f977fe640 1 -- 172.21.15.77:0/1985227502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8018070 con 0x7f2fb015e4c0 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2f977fe640 1 -- 172.21.15.77:0/1985227502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa8007500 con 0x7f2fb015e4c0 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2fb5476640 1 -- 172.21.15.77:0/1985227502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb01605a0 con 0x7f2fb015e4c0 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2f977fe640 1 -- 172.21.15.77:0/1985227502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8004a20 con 0x7f2fb015e4c0 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.010+0000 7f2fb5476640 1 -- 172.21.15.77:0/1985227502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb0160de0 con 0x7f2fb015e4c0 2024-04-25T01:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.011+0000 7f2fb5476640 1 -- 172.21.15.77:0/1985227502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb0068fb0 con 0x7f2fb015e4c0 2024-04-25T01:22:08.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.012+0000 7f2f977fe640 1 -- 172.21.15.77:0/1985227502 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2fa801b070 con 0x7f2fb015e4c0 2024-04-25T01:22:08.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.012+0000 7f2f977fe640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2f9c06be30 0x7f2f9c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:08.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.012+0000 7f2f977fe640 1 -- 172.21.15.77:0/1985227502 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2fa808a3e0 con 0x7f2fb015e4c0 2024-04-25T01:22:08.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.012+0000 7f2faeffd640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2f9c06be30 0x7f2f9c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:08.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.013+0000 7f2faeffd640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2f9c06be30 0x7f2f9c06e2f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2fb01f7390 tx=0x7f2fa00039b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:08.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.017+0000 7f2f977fe640 1 -- 172.21.15.77:0/1985227502 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2fb01605a0 con 0x7f2fb015e4c0 2024-04-25T01:22:09.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.192+0000 7f2fb5476640 1 -- 172.21.15.77:0/1985227502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2fb00008d0 con 0x7f2fb015e4c0 2024-04-25T01:22:09.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.193+0000 7f2f977fe640 1 -- 172.21.15.77:0/1985227502 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f2fb00ccf60 con 0x7f2fb015e4c0 2024-04-25T01:22:09.174 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:09.174 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:09.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 -- 172.21.15.77:0/1985227502 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2f9c06be30 msgr2=0x7f2f9c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2f9c06be30 0x7f2f9c06e2f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2fb01f7390 tx=0x7f2fa00039b0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 -- 172.21.15.77:0/1985227502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb015e4c0 msgr2=0x7f2fb015cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb015e4c0 0x7f2fb015cd30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800c480 tx=0x7f2fa800c940 comp rx=0 tx=0).stop 2024-04-25T01:22:09.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 -- 172.21.15.77:0/1985227502 shutdown_connections 2024-04-25T01:22:09.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2f9c06be30 0x7f2f9c06e2f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb015e4c0 0x7f2fb015cd30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb00d4a10 0x7f2fb0153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 --2- 172.21.15.77:0/1985227502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00d2d40 0x7f2fb00cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 -- 172.21.15.77:0/1985227502 >> 172.21.15.77:0/1985227502 conn(0x7f2fb00d3ef0 msgr2=0x7f2fb00d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 -- 172.21.15.77:0/1985227502 shutdown_connections 2024-04-25T01:22:09.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.195+0000 7f2fb5476640 1 -- 172.21.15.77:0/1985227502 wait complete. 2024-04-25T01:22:09.186 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T01:22:09.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:09.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.303+0000 7fcc1f8bd640 1 Processor -- start 2024-04-25T01:22:09.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.303+0000 7fcc1f8bd640 1 -- start start 2024-04-25T01:22:09.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1f8bd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcc18143940 0x7fcc1806e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1f8bd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc1815f400 0x7fcc1806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1f8bd640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc18070e60 con 0x7fcc18143d20 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1f8bd640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fcc18071140 con 0x7fcc18143940 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1f8bd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcc18071420 con 0x7fcc1815f400 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1d659640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcc18143940 0x7fcc1806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1ce58640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcc18143d20 0x7fcc1815ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56422/0 (socket says 172.21.15.77:56422) 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1ce58640 1 -- 172.21.15.77:0/424849798 learned_addr learned my addr 172.21.15.77:0/424849798 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1d659640 1 --2- 172.21.15.77:0/424849798 >> v2:172.21.15.77:3300/0 conn(0x7fcc18143940 0x7fcc1806e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56072/0 (socket says 172.21.15.77:56072) 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1de5a640 1 --2- 172.21.15.77:0/424849798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc1815f400 0x7fcc1806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1d659640 1 -- 172.21.15.77:0/424849798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc1815f400 msgr2=0x7fcc1806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1d659640 1 --2- 172.21.15.77:0/424849798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc1815f400 0x7fcc1806eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1d659640 1 -- 172.21.15.77:0/424849798 >> v1:172.21.15.77:6789/0 conn(0x7fcc18143d20 legacy=0x7fcc1815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1d659640 1 -- 172.21.15.77:0/424849798 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc18071700 con 0x7fcc18143940 2024-04-25T01:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.304+0000 7fcc1de5a640 1 --2- 172.21.15.77:0/424849798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc1815f400 0x7fcc1806eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.305+0000 7fcc1d659640 1 --2- 172.21.15.77:0/424849798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc18143940 0x7fcc1806e650 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcc140097d0 tx=0x7fcc1402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31826ab48c4debc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.305+0000 7fcc0ffff640 1 -- 172.21.15.77:0/424849798 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14037070 con 0x7fcc18143940 2024-04-25T01:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.305+0000 7fcc0ffff640 1 -- 172.21.15.77:0/424849798 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc14032440 con 0x7fcc18143940 2024-04-25T01:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.305+0000 7fcc0ffff640 1 -- 172.21.15.77:0/424849798 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14032810 con 0x7fcc18143940 2024-04-25T01:22:09.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.305+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/424849798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc18143940 msgr2=0x7fcc1806e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.305+0000 7fcc1f8bd640 1 --2- 172.21.15.77:0/424849798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc18143940 0x7fcc1806e650 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcc140097d0 tx=0x7fcc1402a1c0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.306+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/424849798 shutdown_connections 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.306+0000 7fcc1f8bd640 1 --2- 172.21.15.77:0/424849798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc1815f400 0x7fcc1806eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.306+0000 7fcc1f8bd640 1 --2- 172.21.15.77:0/424849798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc18143940 0x7fcc1806e650 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.306+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/424849798 >> 172.21.15.77:0/424849798 conn(0x7fcc180b1820 msgr2=0x7fcc180b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.306+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/424849798 shutdown_connections 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.306+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/424849798 wait complete. 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.307+0000 7fcc1f8bd640 1 Processor -- start 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.307+0000 7fcc1f8bd640 1 -- start start 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.307+0000 7fcc1f8bd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc18143940 0x7fcc18158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.307+0000 7fcc1f8bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc18143d20 0x7fcc18141c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.307+0000 7fcc1f8bd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc1815f400 0x7fcc18072590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.307+0000 7fcc1f8bd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcc18071fb0 con 0x7fcc18143d20 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.307+0000 7fcc1f8bd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcc18071e30 con 0x7fcc18143940 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.307+0000 7fcc1f8bd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcc18072130 con 0x7fcc1815f400 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.307+0000 7fcc1ce58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc18143d20 0x7fcc18141c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1ce58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc18143d20 0x7fcc18141c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56084/0 (socket says 172.21.15.77:56084) 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1ce58640 1 -- 172.21.15.77:0/1603481942 learned_addr learned my addr 172.21.15.77:0/1603481942 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1d659640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc18143940 0x7fcc18158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1de5a640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc1815f400 0x7fcc18072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1ce58640 1 -- 172.21.15.77:0/1603481942 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc1815f400 msgr2=0x7fcc18072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1ce58640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc1815f400 0x7fcc18072590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1ce58640 1 -- 172.21.15.77:0/1603481942 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc18143940 msgr2=0x7fcc18158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1ce58640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc18143940 0x7fcc18158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1ce58640 1 -- 172.21.15.77:0/1603481942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc18157ab0 con 0x7fcc18143d20 2024-04-25T01:22:09.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1d659640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc18143940 0x7fcc18158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:09.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1ce58640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc18143d20 0x7fcc18141c90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800e710 tx=0x7fcc0800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc0dffb640 1 -- 172.21.15.77:0/1603481942 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc08015070 con 0x7fcc18143d20 2024-04-25T01:22:09.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc0dffb640 1 -- 172.21.15.77:0/1603481942 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc08005ce0 con 0x7fcc18143d20 2024-04-25T01:22:09.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc0dffb640 1 -- 172.21.15.77:0/1603481942 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc08001920 con 0x7fcc18143d20 2024-04-25T01:22:09.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/1603481942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc18157d90 con 0x7fcc18143d20 2024-04-25T01:22:09.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.308+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/1603481942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc18158350 con 0x7fcc18143d20 2024-04-25T01:22:09.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.309+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/1603481942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc18068fb0 con 0x7fcc18143d20 2024-04-25T01:22:09.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.309+0000 7fcc0dffb640 1 -- 172.21.15.77:0/1603481942 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcc08018070 con 0x7fcc18143d20 2024-04-25T01:22:09.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.310+0000 7fcc0dffb640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcbfc06be60 0x7fcbfc06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.310+0000 7fcc0dffb640 1 -- 172.21.15.77:0/1603481942 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcc0808d2a0 con 0x7fcc18143d20 2024-04-25T01:22:09.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.310+0000 7fcc1d659640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcbfc06be60 0x7fcbfc06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.310+0000 7fcc1d659640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcbfc06be60 0x7fcbfc06e320 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcc14034000 tx=0x7fcc1402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.314+0000 7fcc0dffb640 1 -- 172.21.15.77:0/1603481942 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcc0800ee20 con 0x7fcc18143d20 2024-04-25T01:22:09.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.486+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/1603481942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc1806d150 con 0x7fcc18143d20 2024-04-25T01:22:09.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.487+0000 7fcc0dffb640 1 -- 172.21.15.77:0/1603481942 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7fcc0805b9a0 con 0x7fcc18143d20 2024-04-25T01:22:09.468 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:09.468 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:09.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/1603481942 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcbfc06be60 msgr2=0x7fcbfc06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcbfc06be60 0x7fcbfc06e320 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcc14034000 tx=0x7fcc1402e000 comp rx=0 tx=0).stop 2024-04-25T01:22:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/1603481942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc18143d20 msgr2=0x7fcc18141c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc18143d20 0x7fcc18141c90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800e710 tx=0x7fcc0800ebd0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/1603481942 shutdown_connections 2024-04-25T01:22:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcbfc06be60 0x7fcbfc06e320 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc18143d20 0x7fcc18141c90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc1815f400 0x7fcc18072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 --2- 172.21.15.77:0/1603481942 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc18143940 0x7fcc18158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/1603481942 >> 172.21.15.77:0/1603481942 conn(0x7fcc180b1820 msgr2=0x7fcc180cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/1603481942 shutdown_connections 2024-04-25T01:22:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.489+0000 7fcc1f8bd640 1 -- 172.21.15.77:0/1603481942 wait complete. 2024-04-25T01:22:09.482 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T01:22:09.482 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T01:22:09.482 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T01:22:09.483 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T01:22:09.483 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-25T01:22:09.483 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-25T01:22:09.484 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-25T01:22:09.484 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-25T01:22:09.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.669+0000 7f0563a5d640 1 Processor -- start 2024-04-25T01:22:09.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.669+0000 7f0563a5d640 1 -- start start 2024-04-25T01:22:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.670+0000 7f60990d4640 1 Processor -- start 2024-04-25T01:22:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.670+0000 7f0aec808640 1 Processor -- start 2024-04-25T01:22:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.670+0000 7f60990d4640 1 -- start start 2024-04-25T01:22:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.670+0000 7f1ab4e9e640 1 Processor -- start 2024-04-25T01:22:09.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.670+0000 7f0aec808640 1 -- start start 2024-04-25T01:22:09.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.670+0000 7f7c4efe2640 1 Processor -- start 2024-04-25T01:22:09.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.670+0000 7f39b8d7e640 1 Processor -- start 2024-04-25T01:22:09.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.670+0000 7f39b8d7e640 1 -- start start 2024-04-25T01:22:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f0563a5d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f05540e30c0 0x7f0554057d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f6d1aa3c640 1 Processor -- start 2024-04-25T01:22:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f0563a5d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05540582d0 con 0x7f055403e840 2024-04-25T01:22:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.670+0000 7f1ab4e9e640 1 -- start start 2024-04-25T01:22:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f05617f9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f05540e30c0 0x7f0554057d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f0560ff8640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f055403e840 0x7f0554053bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56438/0 (socket says 172.21.15.77:56438) 2024-04-25T01:22:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f0560ff8640 1 -- 172.21.15.77:0/3188806107 learned_addr learned my addr 172.21.15.77:0/3188806107 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f0563a5d640 1 -- 172.21.15.77:0/3188806107 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0554058570 con 0x7f0554054300 2024-04-25T01:22:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f0563a5d640 1 -- 172.21.15.77:0/3188806107 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f0554058850 con 0x7f05540e30c0 2024-04-25T01:22:09.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f60990d4640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6094159ee0 0x7f60940b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f60990d4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60940b4790 0x7f6094070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f6092d9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6094159ee0 0x7f60940b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f6092d9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6094159ee0 0x7f60940b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48662/0 (socket says 172.21.15.77:48662) 2024-04-25T01:22:09.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.670+0000 7f7c4efe2640 1 -- start start 2024-04-25T01:22:09.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f7c4efe2640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c48159350 0x7f7c480ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f7c4efe2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4806b120 0x7f7c4806b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f7c4efe2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7c48165e90 0x7f7c481662e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4cd7e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c48159350 0x7f7c480ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4cd7e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c48159350 0x7f7c480ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48664/0 (socket says 172.21.15.77:48664) 2024-04-25T01:22:09.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f6092d9d640 1 -- 172.21.15.77:0/3854542801 learned_addr learned my addr 172.21.15.77:0/3854542801 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f0aec808640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0ae80cec30 0x7f0ae80cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f0aec808640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ae80b33c0 0x7f0ae80b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4efe2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f7c480cef30 con 0x7f7c48165e90 2024-04-25T01:22:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4d57f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7c48165e90 0x7f7c481662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4d57f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7c48165e90 0x7f7c481662e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56098/0 (socket says 172.21.15.77:56098) 2024-04-25T01:22:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4d57f640 1 -- 172.21.15.77:0/3176479042 learned_addr learned my addr 172.21.15.77:0/3176479042 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f0aec808640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae806d8e0 con 0x7f0ae81657e0 2024-04-25T01:22:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f60990d4640 1 --2- 172.21.15.77:0/3854542801 >> v2:172.21.15.77:3300/0 conn(0x7f609406ebc0 0x7f6094071010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f0553fff640 1 -- 172.21.15.77:0/3188806107 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3201194342 0 0) 0x7f05540582d0 con 0x7f055403e840 2024-04-25T01:22:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f0553fff640 1 -- 172.21.15.77:0/3188806107 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0548003520 con 0x7f055403e840 2024-04-25T01:22:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f1ab4e9e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ab00c6e30 0x7f1ab015a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f1ab4e9e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1ab006b9a0 0x7f1ab006bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f60990d4640 1 -- 172.21.15.77:0/3854542801 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f60940b3c80 con 0x7f609406ebc0 2024-04-25T01:22:09.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f39b8d7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b40b5030 0x7f39b40b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f39b8d7e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f39b40cfb60 0x7f39b40d0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f39b8d7e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39b40d4d30 con 0x7f39b40cf780 2024-04-25T01:22:09.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f39b8d7e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f39b40b4820 con 0x7f39b40cfb60 2024-04-25T01:22:09.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f39b8d7e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f39b40b4260 con 0x7f39b40b5030 2024-04-25T01:22:09.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f1aae59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ab00c6e30 0x7f1ab015a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f0ae6d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0ae81657e0 0x7f0ae806d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56454/0 (socket says 172.21.15.77:56454) 2024-04-25T01:22:09.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f0ae6d9d640 1 -- 172.21.15.77:0/2869829938 learned_addr learned my addr 172.21.15.77:0/2869829938 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:09.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f0ae759e640 1 --2- 172.21.15.77:0/2869829938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ae80b33c0 0x7f0ae80b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f609359e640 1 --2- 172.21.15.77:0/3854542801 >> v2:172.21.15.77:3300/0 conn(0x7f609406ebc0 0x7f6094071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f60990d4640 1 -- 172.21.15.77:0/3854542801 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f60940b39a0 con 0x7f6094159ee0 2024-04-25T01:22:09.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f39b2d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f39b40cfb60 0x7f39b40d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f39b2d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f39b40cfb60 0x7f39b40d0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56124/0 (socket says 172.21.15.77:56124) 2024-04-25T01:22:09.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f39b2d9d640 1 -- 172.21.15.77:0/2513185089 learned_addr learned my addr 172.21.15.77:0/2513185089 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f0aec808640 1 -- 172.21.15.77:0/2869829938 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae80cf600 con 0x7f0ae80cec30 2024-04-25T01:22:09.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f0aec808640 1 -- 172.21.15.77:0/2869829938 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae8165bc0 con 0x7f0ae80b33c0 2024-04-25T01:22:09.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f60990d4640 1 -- 172.21.15.77:0/3854542801 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f609415a450 con 0x7f60940b4790 2024-04-25T01:22:09.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.671+0000 7f6d1aa3c640 1 -- start start 2024-04-25T01:22:09.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f6d1aa3c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6d140cfdf0 0x7f6d140d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f6d1aa3c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d14147a80 0x7f6d14147f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f6d1aa3c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d140d0710 con 0x7f6d14070a80 2024-04-25T01:22:09.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f6d1aa3c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6d14145690 con 0x7f6d14147a80 2024-04-25T01:22:09.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f6d1aa3c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6d14162930 con 0x7f6d140cfdf0 2024-04-25T01:22:09.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f34f742a640 1 Processor -- start 2024-04-25T01:22:09.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f39b2d9d640 1 -- 172.21.15.77:0/2513185089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b40b5030 msgr2=0x7f39b40b49e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:22:09.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f39b2d9d640 1 --2- 172.21.15.77:0/2513185089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b40b5030 0x7f39b40b49e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f39b2d9d640 1 -- 172.21.15.77:0/2513185089 >> v1:172.21.15.77:6789/0 conn(0x7f39b40cf780 legacy=0x7f39b40cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f34f742a640 1 -- start start 2024-04-25T01:22:09.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f39b2d9d640 1 -- 172.21.15.77:0/2513185089 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b40b4540 con 0x7f39b40cfb60 2024-04-25T01:22:09.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f6d19239640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d14147a80 0x7f6d14147f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f6d19239640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d14147a80 0x7f6d14147f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56136/0 (socket says 172.21.15.77:56136) 2024-04-25T01:22:09.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f6d19239640 1 -- 172.21.15.77:0/738406555 learned_addr learned my addr 172.21.15.77:0/738406555 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f609259c640 1 --2- 172.21.15.77:0/3854542801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60940b4790 0x7f6094070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f6d19a3a640 1 --2- 172.21.15.77:0/738406555 >> v2:172.21.15.121:3300/0 conn(0x7f6d140cfdf0 0x7f6d140d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f05617f9640 1 -- 172.21.15.77:0/3188806107 >> v1:172.21.15.121:6789/0 conn(0x7f0554054300 legacy=0x7f05540068d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f05617f9640 1 -- 172.21.15.77:0/3188806107 >> v1:172.21.15.77:6789/0 conn(0x7f055403e840 legacy=0x7f0554053bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f05617f9640 1 -- 172.21.15.77:0/3188806107 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0554058b30 con 0x7f05540e30c0 2024-04-25T01:22:09.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f05617f9640 1 --2- 172.21.15.77:0/3188806107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05540e30c0 0x7f0554057d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f055c050770 tx=0x7f055c0a5df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf99660841c4a7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3176479042 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7c480b4620 con 0x7f7c48159350 2024-04-25T01:22:09.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3176479042 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7c48159730 con 0x7f7c4806b120 2024-04-25T01:22:09.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f39b2d9d640 1 --2- 172.21.15.77:0/2513185089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b40cfb60 0x7f39b40d0580 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f39ac009e50 tx=0x7f39ac02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7566d75f0ff19f21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f6d19a3a640 1 -- 172.21.15.77:0/738406555 >> v2:172.21.15.77:3300/0 conn(0x7f6d14147a80 msgr2=0x7f6d14147f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4d57f640 1 -- 172.21.15.77:0/3176479042 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4806b120 msgr2=0x7f7c4806b5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:22:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f6d19a3a640 1 --2- 172.21.15.77:0/738406555 >> v2:172.21.15.77:3300/0 conn(0x7f6d14147a80 0x7f6d14147f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f6d19a3a640 1 -- 172.21.15.77:0/738406555 >> v1:172.21.15.121:6789/0 conn(0x7f6d14070a80 legacy=0x7f6d14070f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4d57f640 1 --2- 172.21.15.77:0/3176479042 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4806b120 0x7f7c4806b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f6d19a3a640 1 -- 172.21.15.77:0/738406555 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d14162c10 con 0x7f6d140cfdf0 2024-04-25T01:22:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4d57f640 1 -- 172.21.15.77:0/3176479042 >> v2:172.21.15.121:3300/0 conn(0x7f7c48159350 msgr2=0x7f7c480ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4d57f640 1 --2- 172.21.15.77:0/3176479042 >> v2:172.21.15.121:3300/0 conn(0x7f7c48159350 0x7f7c480ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f7c4d57f640 1 -- 172.21.15.77:0/3176479042 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c480b3a10 con 0x7f7c48165e90 2024-04-25T01:22:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f39b159a640 1 -- 172.21.15.77:0/2513185089 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac036070 con 0x7f39b40cfb60 2024-04-25T01:22:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f0ae659c640 1 --2- 172.21.15.77:0/2869829938 >> v2:172.21.15.77:3300/0 conn(0x7f0ae80cec30 0x7f0ae80cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f0ae5d9b640 1 -- 172.21.15.77:0/2869829938 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 567817889 0 0) 0x7f0ae806d8e0 con 0x7f0ae81657e0 2024-04-25T01:22:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f0ae5d9b640 1 -- 172.21.15.77:0/2869829938 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ad0003520 con 0x7f0ae81657e0 2024-04-25T01:22:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f0ae659c640 1 -- 172.21.15.77:0/2869829938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ae80b33c0 msgr2=0x7f0ae80b37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f0ae659c640 1 --2- 172.21.15.77:0/2869829938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ae80b33c0 0x7f0ae80b37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f0ae659c640 1 -- 172.21.15.77:0/2869829938 >> v1:172.21.15.77:6789/0 conn(0x7f0ae81657e0 legacy=0x7f0ae806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f0ae659c640 1 -- 172.21.15.77:0/2869829938 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae80cde70 con 0x7f0ae80cec30 2024-04-25T01:22:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f7c4d57f640 1 --2- 172.21.15.77:0/3176479042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48165e90 0x7f7c481662e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7c480b26d0 tx=0x7f7c4002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ee57cda3d64045 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f7c4cd7e640 1 --2- 172.21.15.77:0/3176479042 >> v2:172.21.15.121:3300/0 conn(0x7f7c48159350 0x7f7c480ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:09.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.672+0000 7f1aadd9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f1ab00d0da0 0x7f1ab0066ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:59600/0 (socket says 172.21.15.77:59600) 2024-04-25T01:22:09.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f1ab4e9e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ab00b5da0 con 0x7f1ab00d0da0 2024-04-25T01:22:09.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f1ab4e9e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab00afe20 con 0x7f1ab00c6e30 2024-04-25T01:22:09.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f1aae59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ab00c6e30 0x7f1ab015a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56130/0 (socket says 172.21.15.77:56130) 2024-04-25T01:22:09.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f1aae59c640 1 -- 172.21.15.77:0/247774753 learned_addr learned my addr 172.21.15.77:0/247774753 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/247774753 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab00675b0 con 0x7f1ab006b9a0 2024-04-25T01:22:09.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f1aae59c640 1 -- 172.21.15.77:0/247774753 >> v2:172.21.15.121:3300/0 conn(0x7f1ab006b9a0 msgr2=0x7f1ab006bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f1aae59c640 1 --2- 172.21.15.77:0/247774753 >> v2:172.21.15.121:3300/0 conn(0x7f1ab006b9a0 0x7f1ab006bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f1aae59c640 1 -- 172.21.15.77:0/247774753 >> v1:172.21.15.121:6789/0 conn(0x7f1ab00d0da0 legacy=0x7f1ab0066ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f39b159a640 1 -- 172.21.15.77:0/2513185089 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39ac031600 con 0x7f39b40cfb60 2024-04-25T01:22:09.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f39b159a640 1 -- 172.21.15.77:0/2513185089 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0319d0 con 0x7f39b40cfb60 2024-04-25T01:22:09.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f1aae59c640 1 -- 172.21.15.77:0/247774753 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab00af6f0 con 0x7f1ab00c6e30 2024-04-25T01:22:09.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f6092d9d640 1 -- 172.21.15.77:0/3854542801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60940b4790 msgr2=0x7f6094070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f6092d9d640 1 --2- 172.21.15.77:0/3854542801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60940b4790 0x7f6094070ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f6092d9d640 1 -- 172.21.15.77:0/3854542801 >> v2:172.21.15.77:3300/0 conn(0x7f609406ebc0 msgr2=0x7f6094071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f6092d9d640 1 --2- 172.21.15.77:0/3854542801 >> v2:172.21.15.77:3300/0 conn(0x7f609406ebc0 0x7f6094071010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.673+0000 7f6092d9d640 1 -- 172.21.15.77:0/3854542801 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609415a730 con 0x7f6094159ee0 2024-04-25T01:22:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f7c477fe640 1 -- 172.21.15.77:0/3176479042 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40036070 con 0x7f7c48165e90 2024-04-25T01:22:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f7c477fe640 1 -- 172.21.15.77:0/3176479042 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c40031600 con 0x7f7c48165e90 2024-04-25T01:22:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f1aadd9b640 1 --1- 172.21.15.77:0/247774753 >> v1:172.21.15.121:6789/0 conn(0x7f1ab00d0da0 0x7f1ab0066ea0 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-25T01:22:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f34f742a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34f00cfdf0 0x7f34f00d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f34f742a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34f0147a80 0x7f34f0147f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f34f742a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f34f0070a80 0x7f34f0070f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f34f742a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f34f0145690 con 0x7f34f0147a80 2024-04-25T01:22:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f34f742a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f34f00d0710 con 0x7f34f0070a80 2024-04-25T01:22:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f34f742a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f34f0162930 con 0x7f34f00cfdf0 2024-04-25T01:22:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f34f6428640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34f00cfdf0 0x7f34f00d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f6d19239640 1 --2- 172.21.15.77:0/738406555 >> v2:172.21.15.77:3300/0 conn(0x7f6d14147a80 0x7f6d14147f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f34f6c29640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f34f0070a80 0x7f34f0070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f34f6c29640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f34f0070a80 0x7f34f0070f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48684/0 (socket says 172.21.15.77:48684) 2024-04-25T01:22:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f34f6c29640 1 -- 172.21.15.77:0/1076351001 learned_addr learned my addr 172.21.15.77:0/1076351001 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.674+0000 7f7c477fe640 1 -- 172.21.15.77:0/3176479042 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c400319b0 con 0x7f7c48165e90 2024-04-25T01:22:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f0ae659c640 1 --2- 172.21.15.77:0/2869829938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae80cec30 0x7f0ae80cf0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0097a0 tx=0x7f0adc02a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8017b236270d748b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f34f6c29640 1 -- 172.21.15.77:0/1076351001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34f00cfdf0 msgr2=0x7f34f00d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f34f6c29640 1 --2- 172.21.15.77:0/1076351001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34f00cfdf0 0x7f34f00d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f34f6c29640 1 -- 172.21.15.77:0/1076351001 >> v2:172.21.15.77:3300/0 conn(0x7f34f0147a80 msgr2=0x7f34f0147f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f34f6c29640 1 --2- 172.21.15.77:0/1076351001 >> v2:172.21.15.77:3300/0 conn(0x7f34f0147a80 0x7f34f0147f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f34f6c29640 1 -- 172.21.15.77:0/1076351001 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f0162c10 con 0x7f34f0070a80 2024-04-25T01:22:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f6092d9d640 1 --2- 172.21.15.77:0/3854542801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6094159ee0 0x7f60940b40b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f60940689c0 tx=0x7f608802b1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1696624d7bc7726 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f0553fff640 1 -- 172.21.15.77:0/3188806107 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0b3070 con 0x7f05540e30c0 2024-04-25T01:22:09.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f0553fff640 1 -- 172.21.15.77:0/3188806107 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f055c0a6bc0 con 0x7f05540e30c0 2024-04-25T01:22:09.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f0553fff640 1 -- 172.21.15.77:0/3188806107 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0ae440 con 0x7f05540e30c0 2024-04-25T01:22:09.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f6d19a3a640 1 --2- 172.21.15.77:0/738406555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d140cfdf0 0x7f6d140d01d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d100097a0 tx=0x7f6d1002a0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3632fdeec2af6efb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3176479042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48165e90 msgr2=0x7f7c481662e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f7c4efe2640 1 --2- 172.21.15.77:0/3176479042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48165e90 0x7f7c481662e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7c480b26d0 tx=0x7f7c4002a6d0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3176479042 shutdown_connections 2024-04-25T01:22:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f7c4efe2640 1 --2- 172.21.15.77:0/3176479042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48165e90 0x7f7c481662e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f7c4efe2640 1 --2- 172.21.15.77:0/3176479042 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4806b120 0x7f7c4806b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f7c4efe2640 1 --2- 172.21.15.77:0/3176479042 >> v2:172.21.15.121:3300/0 conn(0x7f7c48159350 0x7f7c480ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3176479042 >> 172.21.15.77:0/3176479042 conn(0x7f7c480b1660 msgr2=0x7f7c480b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f39b8d7e640 1 -- 172.21.15.77:0/2513185089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b40cfb60 msgr2=0x7f39b40d0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f39b8d7e640 1 --2- 172.21.15.77:0/2513185089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b40cfb60 0x7f39b40d0580 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f39ac009e50 tx=0x7f39ac02a6f0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f6d18a38640 1 -- 172.21.15.77:0/738406555 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10037070 con 0x7f6d140cfdf0 2024-04-25T01:22:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3176479042 shutdown_connections 2024-04-25T01:22:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f0ae5d9b640 1 -- 172.21.15.77:0/2869829938 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc037070 con 0x7f0ae80cec30 2024-04-25T01:22:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f0ae5d9b640 1 -- 172.21.15.77:0/2869829938 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0adc02ae40 con 0x7f0ae80cec30 2024-04-25T01:22:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f0ae5d9b640 1 -- 172.21.15.77:0/2869829938 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc032660 con 0x7f0ae80cec30 2024-04-25T01:22:09.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f6d18a38640 1 -- 172.21.15.77:0/738406555 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d1002ad70 con 0x7f6d140cfdf0 2024-04-25T01:22:09.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.675+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3176479042 wait complete. 2024-04-25T01:22:09.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f6d18a38640 1 -- 172.21.15.77:0/738406555 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10032660 con 0x7f6d140cfdf0 2024-04-25T01:22:09.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f39b8d7e640 1 -- 172.21.15.77:0/2513185089 shutdown_connections 2024-04-25T01:22:09.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f39b8d7e640 1 --2- 172.21.15.77:0/2513185089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b40cfb60 0x7f39b40d0580 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f39b8d7e640 1 --2- 172.21.15.77:0/2513185089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b40b5030 0x7f39b40b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f39b8d7e640 1 -- 172.21.15.77:0/2513185089 >> 172.21.15.77:0/2513185089 conn(0x7f39b40d3e50 msgr2=0x7f39b40d4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f34f6c29640 1 --2- 172.21.15.77:0/1076351001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f0070a80 0x7f34f0070f10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f34f00d09a0 tx=0x7f34e002a1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66889394870157d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f6091d9b640 1 -- 172.21.15.77:0/3854542801 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60880089e0 con 0x7f6094159ee0 2024-04-25T01:22:09.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f0563a5d640 1 -- 172.21.15.77:0/3188806107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05540e30c0 msgr2=0x7f0554057d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f0563a5d640 1 --2- 172.21.15.77:0/3188806107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05540e30c0 0x7f0554057d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f055c050770 tx=0x7f055c0a5df0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f0aec808640 1 -- 172.21.15.77:0/2869829938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae80cec30 msgr2=0x7f0ae80cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f0aec808640 1 --2- 172.21.15.77:0/2869829938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae80cec30 0x7f0ae80cf0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0097a0 tx=0x7f0adc02a0f0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f6091d9b640 1 -- 172.21.15.77:0/3854542801 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6088008b80 con 0x7f6094159ee0 2024-04-25T01:22:09.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f6091d9b640 1 -- 172.21.15.77:0/3854542801 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608800a0d0 con 0x7f6094159ee0 2024-04-25T01:22:09.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f39b8d7e640 1 -- 172.21.15.77:0/2513185089 shutdown_connections 2024-04-25T01:22:09.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/738406555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d140cfdf0 msgr2=0x7f6d140d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f34f5426640 1 -- 172.21.15.77:0/1076351001 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0037070 con 0x7f34f0070a80 2024-04-25T01:22:09.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f0563a5d640 1 -- 172.21.15.77:0/3188806107 shutdown_connections 2024-04-25T01:22:09.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f0563a5d640 1 --2- 172.21.15.77:0/3188806107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05540e30c0 0x7f0554057d90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f0563a5d640 1 -- 172.21.15.77:0/3188806107 >> 172.21.15.77:0/3188806107 conn(0x7f05540559e0 msgr2=0x7f0554055df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f0aec808640 1 -- 172.21.15.77:0/2869829938 shutdown_connections 2024-04-25T01:22:09.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f0aec808640 1 --2- 172.21.15.77:0/2869829938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ae80b33c0 0x7f0ae80b37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f0aec808640 1 --2- 172.21.15.77:0/2869829938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae80cec30 0x7f0ae80cf0c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f0aec808640 1 -- 172.21.15.77:0/2869829938 >> 172.21.15.77:0/2869829938 conn(0x7f0ae806bad0 msgr2=0x7f0ae80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f34f5426640 1 -- 172.21.15.77:0/1076351001 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e0032440 con 0x7f34f0070a80 2024-04-25T01:22:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f0aec808640 1 -- 172.21.15.77:0/2869829938 shutdown_connections 2024-04-25T01:22:09.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f0aec808640 1 -- 172.21.15.77:0/2869829938 wait complete. 2024-04-25T01:22:09.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f6d1aa3c640 1 --2- 172.21.15.77:0/738406555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d140cfdf0 0x7f6d140d01d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d100097a0 tx=0x7f6d1002a0e0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/738406555 shutdown_connections 2024-04-25T01:22:09.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f6d1aa3c640 1 --2- 172.21.15.77:0/738406555 >> v2:172.21.15.77:3300/0 conn(0x7f6d14147a80 0x7f6d14147f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f6d1aa3c640 1 --2- 172.21.15.77:0/738406555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d140cfdf0 0x7f6d140d01d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/738406555 >> 172.21.15.77:0/738406555 conn(0x7f6d1406b540 msgr2=0x7f6d1406b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f7c4efe2640 1 Processor -- start 2024-04-25T01:22:09.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.676+0000 7f39b8d7e640 1 -- 172.21.15.77:0/2513185089 wait complete. 2024-04-25T01:22:09.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f7c4efe2640 1 -- start start 2024-04-25T01:22:09.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f0563a5d640 1 -- 172.21.15.77:0/3188806107 shutdown_connections 2024-04-25T01:22:09.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f0563a5d640 1 -- 172.21.15.77:0/3188806107 wait complete. 2024-04-25T01:22:09.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f60990d4640 1 -- 172.21.15.77:0/3854542801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6094159ee0 msgr2=0x7f60940b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f60990d4640 1 --2- 172.21.15.77:0/3854542801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6094159ee0 0x7f60940b40b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f60940689c0 tx=0x7f608802b1f0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f60990d4640 1 -- 172.21.15.77:0/3854542801 shutdown_connections 2024-04-25T01:22:09.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f60990d4640 1 --2- 172.21.15.77:0/3854542801 >> v2:172.21.15.77:3300/0 conn(0x7f609406ebc0 0x7f6094071010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f60990d4640 1 --2- 172.21.15.77:0/3854542801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60940b4790 0x7f6094070ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f60990d4640 1 --2- 172.21.15.77:0/3854542801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6094159ee0 0x7f60940b40b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f60990d4640 1 -- 172.21.15.77:0/3854542801 >> 172.21.15.77:0/3854542801 conn(0x7f609406b300 msgr2=0x7f609406b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f34f5426640 1 -- 172.21.15.77:0/1076351001 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0032810 con 0x7f34f0070a80 2024-04-25T01:22:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f34f742a640 1 -- 172.21.15.77:0/1076351001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f0070a80 msgr2=0x7f34f0070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f34f742a640 1 --2- 172.21.15.77:0/1076351001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f0070a80 0x7f34f0070f10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f34f00d09a0 tx=0x7f34e002a1f0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.677+0000 7f1aae59c640 1 --2- 172.21.15.77:0/247774753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ab00c6e30 0x7f1ab015a6c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00b2fa0 tx=0x7f1aa4032890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219363b0209e869d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f60990d4640 1 -- 172.21.15.77:0/3854542801 shutdown_connections 2024-04-25T01:22:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f60990d4640 1 -- 172.21.15.77:0/3854542801 wait complete. 2024-04-25T01:22:09.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/738406555 shutdown_connections 2024-04-25T01:22:09.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/738406555 wait complete. 2024-04-25T01:22:09.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f34f742a640 1 -- 172.21.15.77:0/1076351001 shutdown_connections 2024-04-25T01:22:09.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f34f742a640 1 --2- 172.21.15.77:0/1076351001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f0070a80 0x7f34f0070f10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f34f742a640 1 --2- 172.21.15.77:0/1076351001 >> v2:172.21.15.77:3300/0 conn(0x7f34f0147a80 0x7f34f0147f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f34f742a640 1 --2- 172.21.15.77:0/1076351001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34f00cfdf0 0x7f34f00d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f34f742a640 1 -- 172.21.15.77:0/1076351001 >> 172.21.15.77:0/1076351001 conn(0x7f34f006b540 msgr2=0x7f34f006b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f1aad59a640 1 -- 172.21.15.77:0/247774753 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa403f070 con 0x7f1ab00c6e30 2024-04-25T01:22:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f1aad59a640 1 -- 172.21.15.77:0/247774753 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa4036050 con 0x7f1ab00c6e30 2024-04-25T01:22:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f1aad59a640 1 -- 172.21.15.77:0/247774753 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4038970 con 0x7f1ab00c6e30 2024-04-25T01:22:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f39b8d7e640 1 Processor -- start 2024-04-25T01:22:09.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f0563a5d640 1 Processor -- start 2024-04-25T01:22:09.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f0563a5d640 1 -- start start 2024-04-25T01:22:09.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f7c4efe2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4806b120 0x7f7c48151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f34f742a640 1 -- 172.21.15.77:0/1076351001 shutdown_connections 2024-04-25T01:22:09.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f0aec808640 1 Processor -- start 2024-04-25T01:22:09.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f0aec808640 1 -- start start 2024-04-25T01:22:09.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f7c4efe2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c48159350 0x7f7c481fb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f7c4efe2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48165e90 0x7f7c480710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f6d1aa3c640 1 Processor -- start 2024-04-25T01:22:09.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f7c4cd7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4806b120 0x7f7c48151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f7c4cd7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4806b120 0x7f7c48151cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33704/0 (socket says 172.21.15.77:33704) 2024-04-25T01:22:09.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f6d1aa3c640 1 -- start start 2024-04-25T01:22:09.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.678+0000 7f39b8d7e640 1 -- start start 2024-04-25T01:22:09.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0563a5d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f055403e840 0x7f05540e1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c4d57f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48165e90 0x7f7c480710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f34f742a640 1 -- 172.21.15.77:0/1076351001 wait complete. 2024-04-25T01:22:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0563a5d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0554054300 0x7f05541800f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c4d57f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48165e90 0x7f7c480710c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56174/0 (socket says 172.21.15.77:56174) 2024-04-25T01:22:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c4cd7e640 1 -- 172.21.15.77:0/3622683010 learned_addr learned my addr 172.21.15.77:0/3622683010 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f39b8d7e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39b40b5030 0x7f39b40cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f39b8d7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b40cf780 0x7f39b40d3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f39b8d7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b40cfb60 0x7f39b414e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3622683010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480b42b0 con 0x7f7c48165e90 2024-04-25T01:22:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b8d7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f39b40d4af0 con 0x7f39b40cf780 2024-04-25T01:22:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3622683010 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480b3fd0 con 0x7f7c48159350 2024-04-25T01:22:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3622683010 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7c480ce370 con 0x7f7c4806b120 2024-04-25T01:22:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b8d7e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f39b40d4970 con 0x7f39b40b5030 2024-04-25T01:22:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b8d7e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f39b40d0ac0 con 0x7f39b40cfb60 2024-04-25T01:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/247774753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ab00c6e30 msgr2=0x7f1ab015a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.679+0000 7f1ab4e9e640 1 --2- 172.21.15.77:0/247774753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ab00c6e30 0x7f1ab015a6c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00b2fa0 tx=0x7f1aa4032890 comp rx=0 tx=0).stop 2024-04-25T01:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b40cf780 0x7f39b40d3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b40cf780 0x7f39b40d3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56186/0 (socket says 172.21.15.77:56186) 2024-04-25T01:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b1d9b640 1 -- 172.21.15.77:0/855530884 learned_addr learned my addr 172.21.15.77:0/855530884 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b259c640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39b40b5030 0x7f39b40cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f05617f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f055403e840 0x7f05540e1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b2d9d640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b40cfb60 0x7f39b414e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0aec808640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ae80b33c0 0x7f0ae81f58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f60990d4640 1 Processor -- start 2024-04-25T01:22:09.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f60990d4640 1 -- start start 2024-04-25T01:22:09.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0563a5d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05540e30c0 0x7f0554180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0563a5d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05540e34a0 con 0x7f0554054300 2024-04-25T01:22:09.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0563a5d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f055403bfc0 con 0x7f05540e30c0 2024-04-25T01:22:09.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0563a5d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0554006fe0 con 0x7f055403e840 2024-04-25T01:22:09.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f05617f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f055403e840 0x7f05540e1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33720/0 (socket says 172.21.15.77:33720) 2024-04-25T01:22:09.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f05617f9640 1 -- 172.21.15.77:0/828958658 learned_addr learned my addr 172.21.15.77:0/828958658 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c47fff640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c48159350 0x7f7c481fb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0aec808640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae80cec30 0x7f0ae8156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0aec808640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae81657e0 0x7f0ae80b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c4d57f640 1 -- 172.21.15.77:0/3622683010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4806b120 msgr2=0x7f7c48151cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c4d57f640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4806b120 0x7f7c48151cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c4d57f640 1 -- 172.21.15.77:0/3622683010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c48159350 msgr2=0x7f7c481fb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c4d57f640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c48159350 0x7f7c481fb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f7c4d57f640 1 -- 172.21.15.77:0/3622683010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4815adb0 con 0x7f7c48165e90 2024-04-25T01:22:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0ae6d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ae80b33c0 0x7f0ae81f58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f0ae6d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ae80b33c0 0x7f0ae81f58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33736/0 (socket says 172.21.15.77:33736) 2024-04-25T01:22:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0560ff8640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0554054300 0x7f05541800f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f0561ffa640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05540e30c0 0x7f0554180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/247774753 shutdown_connections 2024-04-25T01:22:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f1ab4e9e640 1 --2- 172.21.15.77:0/247774753 >> v2:172.21.15.121:3300/0 conn(0x7f1ab006b9a0 0x7f1ab006bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f1ab4e9e640 1 --2- 172.21.15.77:0/247774753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ab00c6e30 0x7f1ab015a6c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/247774753 >> 172.21.15.77:0/247774753 conn(0x7f1ab00d02b0 msgr2=0x7f1ab00d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/247774753 shutdown_connections 2024-04-25T01:22:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/247774753 wait complete. 2024-04-25T01:22:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f0ae6d9d640 1 -- 172.21.15.77:0/3464669368 learned_addr learned my addr 172.21.15.77:0/3464669368 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f0aec808640 1 -- 172.21.15.77:0/3464669368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae80ce710 con 0x7f0ae80cec30 2024-04-25T01:22:09.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f0ae759e640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae81657e0 0x7f0ae80b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f0aec808640 1 -- 172.21.15.77:0/3464669368 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae80ce430 con 0x7f0ae81657e0 2024-04-25T01:22:09.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f05617f9640 1 -- 172.21.15.77:0/828958658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05540e30c0 msgr2=0x7f0554180630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f05617f9640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05540e30c0 0x7f0554180630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f05617f9640 1 -- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0554054300 msgr2=0x7f05541800f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f05617f9640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0554054300 0x7f05541800f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f05617f9640 1 -- 172.21.15.77:0/828958658 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05540e16d0 con 0x7f055403e840 2024-04-25T01:22:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b2d9d640 1 -- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39b40b5030 msgr2=0x7f39b40cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b2d9d640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39b40b5030 0x7f39b40cc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b2d9d640 1 -- 172.21.15.77:0/855530884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b40cf780 msgr2=0x7f39b40d3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b2d9d640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b40cf780 0x7f39b40d3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.680+0000 7f39b2d9d640 1 -- 172.21.15.77:0/855530884 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b40b8540 con 0x7f39b40cfb60 2024-04-25T01:22:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f39b259c640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39b40b5030 0x7f39b40cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f39b2d9d640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b40cfb60 0x7f39b414e3b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f39ac030000 tx=0x7f39ac004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f0aec808640 1 -- 172.21.15.77:0/3464669368 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae8165e60 con 0x7f0ae80b33c0 2024-04-25T01:22:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f0ae659c640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae80cec30 0x7f0ae8156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f05617f9640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f055403e840 0x7f05540e1b10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f055c06ac00 tx=0x7f055c06b050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f0551ffb640 1 -- 172.21.15.77:0/828958658 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0b3040 con 0x7f055403e840 2024-04-25T01:22:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f0551ffb640 1 -- 172.21.15.77:0/828958658 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f055c0a69c0 con 0x7f055403e840 2024-04-25T01:22:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f39a2ffd640 1 -- 172.21.15.77:0/855530884 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac041020 con 0x7f39b40cfb60 2024-04-25T01:22:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f39a2ffd640 1 -- 172.21.15.77:0/855530884 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39ac0086b0 con 0x7f39b40cfb60 2024-04-25T01:22:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f39a2ffd640 1 -- 172.21.15.77:0/855530884 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac036040 con 0x7f39b40cfb60 2024-04-25T01:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f6d1aa3c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d14070a80 0x7f6d141342f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f6d1aa3c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d140cfdf0 0x7f6d14144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f6d1aa3c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d14147a80 0x7f6d14144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f6d1aa3c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d141636f0 con 0x7f6d14070a80 2024-04-25T01:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f6d1aa3c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6d141631d0 con 0x7f6d14147a80 2024-04-25T01:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f6d1aa3c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6d141487f0 con 0x7f6d140cfdf0 2024-04-25T01:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f6d19a3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d14070a80 0x7f6d141342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f6d19a3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d14070a80 0x7f6d141342f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56214/0 (socket says 172.21.15.77:56214) 2024-04-25T01:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f6d19a3a640 1 -- 172.21.15.77:0/3462736360 learned_addr learned my addr 172.21.15.77:0/3462736360 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f6d19239640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d140cfdf0 0x7f6d14144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f34f742a640 1 Processor -- start 2024-04-25T01:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f0551ffb640 1 -- 172.21.15.77:0/828958658 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0ad950 con 0x7f055403e840 2024-04-25T01:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f0563a5d640 1 -- 172.21.15.77:0/828958658 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05540e4de0 con 0x7f055403e840 2024-04-25T01:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f0563a5d640 1 -- 172.21.15.77:0/828958658 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f05540e0910 con 0x7f055403e840 2024-04-25T01:22:09.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f34f742a640 1 -- start start 2024-04-25T01:22:09.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f39b8d7e640 1 -- 172.21.15.77:0/855530884 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b40b8820 con 0x7f39b40cfb60 2024-04-25T01:22:09.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f39b8d7e640 1 -- 172.21.15.77:0/855530884 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b40b8de0 con 0x7f39b40cfb60 2024-04-25T01:22:09.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f60990d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f609406ebc0 0x7f60940add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f60990d4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60940b4790 0x7f6094153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f60990d4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6094159ee0 0x7f60940b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f60990d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f609406ca80 con 0x7f609406ebc0 2024-04-25T01:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f60990d4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f60940b3670 con 0x7f6094159ee0 2024-04-25T01:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f60990d4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f609406cc00 con 0x7f60940b4790 2024-04-25T01:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f0ae659c640 1 -- 172.21.15.77:0/3464669368 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ae80b33c0 msgr2=0x7f0ae81f58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f0ae659c640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ae80b33c0 0x7f0ae81f58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f0ae659c640 1 -- 172.21.15.77:0/3464669368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae81657e0 msgr2=0x7f0ae80b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f0ae659c640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae81657e0 0x7f0ae80b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f0ae659c640 1 -- 172.21.15.77:0/3464669368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae8157400 con 0x7f0ae80cec30 2024-04-25T01:22:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f7c4d57f640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48165e90 0x7f7c480710c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7c40030000 tx=0x7f7c40004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.681+0000 7f7c4cd7e640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4806b120 0x7f7c48151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f7c457fa640 1 -- 172.21.15.77:0/3622683010 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40040020 con 0x7f7c48165e90 2024-04-25T01:22:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f7c457fa640 1 -- 172.21.15.77:0/3622683010 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c40004040 con 0x7f7c48165e90 2024-04-25T01:22:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f6d1a23b640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d14147a80 0x7f6d14144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f6d19a3a640 1 -- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d140cfdf0 msgr2=0x7f6d14144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f6d19a3a640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d140cfdf0 0x7f6d14144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f6d19a3a640 1 -- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d14147a80 msgr2=0x7f6d14144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f6d19a3a640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d14147a80 0x7f6d14144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f6d19a3a640 1 -- 172.21.15.77:0/3462736360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d1415a080 con 0x7f6d14070a80 2024-04-25T01:22:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f6d1a23b640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d14147a80 0x7f6d14144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f6d19239640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d140cfdf0 0x7f6d14144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f6092d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f609406ebc0 0x7f60940add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f6092d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f609406ebc0 0x7f60940add70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56230/0 (socket says 172.21.15.77:56230) 2024-04-25T01:22:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f6092d9d640 1 -- 172.21.15.77:0/4246505091 learned_addr learned my addr 172.21.15.77:0/4246505091 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.682+0000 7f7c457fa640 1 -- 172.21.15.77:0/3622683010 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40036040 con 0x7f7c48165e90 2024-04-25T01:22:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f34f742a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f0070a80 0x7f34f015ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f34f742a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34f00cfdf0 0x7f34f015f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f34f742a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f0147a80 0x7f34f0148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f34f742a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01636f0 con 0x7f34f0070a80 2024-04-25T01:22:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f34f742a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01631d0 con 0x7f34f0147a80 2024-04-25T01:22:09.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f34f742a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f34f01487f0 con 0x7f34f00cfdf0 2024-04-25T01:22:09.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f609359e640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6094159ee0 0x7f60940b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3622683010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4815b090 con 0x7f7c48165e90 2024-04-25T01:22:09.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3622683010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4815b650 con 0x7f7c48165e90 2024-04-25T01:22:09.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f39b8d7e640 1 -- 172.21.15.77:0/855530884 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f39b4068180 con 0x7f39b40cfb60 2024-04-25T01:22:09.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f34f6c29640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f0147a80 0x7f34f0148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f34f6c29640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f0147a80 0x7f34f0148a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48732/0 (socket says 172.21.15.77:48732) 2024-04-25T01:22:09.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f34f6c29640 1 -- 172.21.15.77:0/1371674193 learned_addr learned my addr 172.21.15.77:0/1371674193 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f34f6428640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f0070a80 0x7f34f015ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f0ae659c640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae80cec30 0x7f0ae8156a20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0adc031000 tx=0x7f0adc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f6d19a3a640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d14070a80 0x7f6d141342f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6d10035040 tx=0x7f6d100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f34f5c27640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34f00cfdf0 0x7f34f015f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f1ab4e9e640 1 Processor -- start 2024-04-25T01:22:09.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f1ab4e9e640 1 -- start start 2024-04-25T01:22:09.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f609259c640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60940b4790 0x7f6094153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f6092d9d640 1 -- 172.21.15.77:0/4246505091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60940b4790 msgr2=0x7f6094153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f6092d9d640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60940b4790 0x7f6094153770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f6092d9d640 1 -- 172.21.15.77:0/4246505091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6094159ee0 msgr2=0x7f60940b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.683+0000 7f6092d9d640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6094159ee0 0x7f60940b1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f6092d9d640 1 -- 172.21.15.77:0/4246505091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60940ae2b0 con 0x7f609406ebc0 2024-04-25T01:22:09.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f6092d9d640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f609406ebc0 0x7f60940add70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6088000c00 tx=0x7f60880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f0563a5d640 1 -- 172.21.15.77:0/828958658 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0554011ef0 con 0x7f055403e840 2024-04-25T01:22:09.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f34f6c29640 1 -- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34f00cfdf0 msgr2=0x7f34f015f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f34f6c29640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34f00cfdf0 0x7f34f015f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f34f6c29640 1 -- 172.21.15.77:0/1371674193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f0070a80 msgr2=0x7f34f015ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f34f6c29640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f0070a80 0x7f34f015ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f34f6c29640 1 -- 172.21.15.77:0/1371674193 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f0149aa0 con 0x7f34f0147a80 2024-04-25T01:22:09.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f34f5c27640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34f00cfdf0 0x7f34f015f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:09.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f607b7fe640 1 -- 172.21.15.77:0/4246505091 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088049020 con 0x7f609406ebc0 2024-04-25T01:22:09.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f39a2ffd640 1 -- 172.21.15.77:0/855530884 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f39ac044020 con 0x7f39b40cfb60 2024-04-25T01:22:09.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f607b7fe640 1 -- 172.21.15.77:0/4246505091 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60880078d0 con 0x7f609406ebc0 2024-04-25T01:22:09.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f607b7fe640 1 -- 172.21.15.77:0/4246505091 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088045e60 con 0x7f609406ebc0 2024-04-25T01:22:09.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f6d0a7fc640 1 -- 172.21.15.77:0/3462736360 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10041030 con 0x7f6d14070a80 2024-04-25T01:22:09.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f6d0a7fc640 1 -- 172.21.15.77:0/3462736360 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d1002ab70 con 0x7f6d14070a80 2024-04-25T01:22:09.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f6d0a7fc640 1 -- 172.21.15.77:0/3462736360 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10037040 con 0x7f6d14070a80 2024-04-25T01:22:09.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/3462736360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d1415a3c0 con 0x7f6d14070a80 2024-04-25T01:22:09.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f60990d4640 1 -- 172.21.15.77:0/4246505091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60941f2b70 con 0x7f609406ebc0 2024-04-25T01:22:09.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.685+0000 7f60990d4640 1 -- 172.21.15.77:0/4246505091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60941f3130 con 0x7f609406ebc0 2024-04-25T01:22:09.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/3462736360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d1415ac00 con 0x7f6d14070a80 2024-04-25T01:22:09.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.685+0000 7f39a2ffd640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f399406bf00 0x7f399406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.685+0000 7f39a2ffd640 1 -- 172.21.15.77:0/855530884 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f39ac0ac770 con 0x7f39b40cfb60 2024-04-25T01:22:09.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.685+0000 7f39a2ffd640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] conn(0x7f3994075680 0x7f3994077ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.685+0000 7f39a2ffd640 1 -- 172.21.15.77:0/855530884 --> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39ac032e50 con 0x7f3994075680 2024-04-25T01:22:09.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.685+0000 7f39a2ffd640 1 -- 172.21.15.77:0/855530884 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f39ac0acb60 con 0x7f39b40cfb60 2024-04-25T01:22:09.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.685+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3622683010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7c48075350 con 0x7f7c48165e90 2024-04-25T01:22:09.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.685+0000 7f7c457fa640 1 -- 172.21.15.77:0/3622683010 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7c40043020 con 0x7f7c48165e90 2024-04-25T01:22:09.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/3462736360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d14068fb0 con 0x7f6d14070a80 2024-04-25T01:22:09.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.685+0000 7f34f6c29640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f0147a80 0x7f34f0148a90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f34e002a230 tx=0x7f34e00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f1ab4e9e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ab006b9a0 0x7f1ab0156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.684+0000 7f1ab4e9e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ab00c6e30 0x7f1ab00ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f1ab4e9e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ab00d0da0 0x7f1ab00c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f1ab4e9e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab006c310 con 0x7f1ab00c6e30 2024-04-25T01:22:09.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f1ab4e9e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab00af4a0 con 0x7f1ab006b9a0 2024-04-25T01:22:09.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f1ab4e9e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab00b5a70 con 0x7f1ab00d0da0 2024-04-25T01:22:09.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f1aadd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ab00c6e30 0x7f1ab00ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f0acf7fe640 1 -- 172.21.15.77:0/3464669368 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc042020 con 0x7f0ae80cec30 2024-04-25T01:22:09.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f0acf7fe640 1 -- 172.21.15.77:0/3464669368 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0adc03d470 con 0x7f0ae80cec30 2024-04-25T01:22:09.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f0acf7fe640 1 -- 172.21.15.77:0/3464669368 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc037040 con 0x7f0ae80cec30 2024-04-25T01:22:09.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f60797fa640 1 -- 172.21.15.77:0/4246505091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6094068fb0 con 0x7f609406ebc0 2024-04-25T01:22:09.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f7c457fa640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c3406bf00 0x7f7c3406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f7c4cd7e640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c3406bf00 0x7f7c3406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f39b259c640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f399406bf00 0x7f399406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f34e6ffd640 1 -- 172.21.15.77:0/1371674193 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0037040 con 0x7f34f0147a80 2024-04-25T01:22:09.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f34e6ffd640 1 -- 172.21.15.77:0/1371674193 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e0042070 con 0x7f34f0147a80 2024-04-25T01:22:09.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f34e6ffd640 1 -- 172.21.15.77:0/1371674193 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0037040 con 0x7f34f0147a80 2024-04-25T01:22:09.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f39b259c640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f399406bf00 0x7f399406e3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f39a40018b0 tx=0x7f39a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f34f742a640 1 -- 172.21.15.77:0/1371674193 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f0149200 con 0x7f34f0147a80 2024-04-25T01:22:09.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f1aadd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ab00c6e30 0x7f1ab00ad1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56252/0 (socket says 172.21.15.77:56252) 2024-04-25T01:22:09.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f1aae59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ab006b9a0 0x7f1ab0156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f1aadd9b640 1 -- 172.21.15.77:0/3134658623 learned_addr learned my addr 172.21.15.77:0/3134658623 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:09.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f0aec808640 1 -- 172.21.15.77:0/3464669368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae81576e0 con 0x7f0ae80cec30 2024-04-25T01:22:09.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f0aec808640 1 -- 172.21.15.77:0/3464669368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae81518c0 con 0x7f0ae80cec30 2024-04-25T01:22:09.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f39b1d9b640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] conn(0x7f3994075680 0x7f3994077ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f7c457fa640 1 -- 172.21.15.77:0/3622683010 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7c400abca0 con 0x7f7c48165e90 2024-04-25T01:22:09.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f7c4cd7e640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c3406bf00 0x7f7c3406e3c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7c4806e7b0 tx=0x7f7c38008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f607b7fe640 1 -- 172.21.15.77:0/4246505091 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f608804c020 con 0x7f609406ebc0 2024-04-25T01:22:09.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f34f742a640 1 -- 172.21.15.77:0/1371674193 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f01497c0 con 0x7f34f0147a80 2024-04-25T01:22:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f7c457fa640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] conn(0x7f7c34075600 0x7f7c34077a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f7c457fa640 1 -- 172.21.15.77:0/3622683010 --> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c4003be50 con 0x7f7c34075600 2024-04-25T01:22:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f1aaed9d640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ab00d0da0 0x7f1ab00c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f1aadd9b640 1 -- 172.21.15.77:0/3134658623 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ab00d0da0 msgr2=0x7f1ab00c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f1aadd9b640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ab00d0da0 0x7f1ab00c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f1aadd9b640 1 -- 172.21.15.77:0/3134658623 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ab006b9a0 msgr2=0x7f1ab0156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f1aadd9b640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ab006b9a0 0x7f1ab0156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.688+0000 7f1aadd9b640 1 -- 172.21.15.77:0/3134658623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab00ada30 con 0x7f1ab00c6e30 2024-04-25T01:22:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.688+0000 7f607b7fe640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f608006be30 0x7f608006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f7c47fff640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] conn(0x7f7c34075600 0x7f7c34077a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.688+0000 7f39b1d9b640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] conn(0x7f3994075680 0x7f3994077ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f0551ffb640 1 -- 172.21.15.77:0/828958658 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f055c0c2030 con 0x7f055403e840 2024-04-25T01:22:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f0551ffb640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f054006be80 0x7f054006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.686+0000 7f0551ffb640 1 -- 172.21.15.77:0/828958658 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f055c127580 con 0x7f055403e840 2024-04-25T01:22:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f0551ffb640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] conn(0x7f0540075600 0x7f0540077a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f0551ffb640 1 -- 172.21.15.77:0/828958658 --> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f055c06d100 con 0x7f0540075600 2024-04-25T01:22:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f0551ffb640 1 -- 172.21.15.77:0/828958658 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f055c127970 con 0x7f055403e840 2024-04-25T01:22:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f0561ffa640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] conn(0x7f0540075600 0x7f0540077a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f0560ff8640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f054006be80 0x7f054006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.687+0000 7f0aec808640 1 -- 172.21.15.77:0/3464669368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0ae8068180 con 0x7f0ae80cec30 2024-04-25T01:22:09.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.688+0000 7f0561ffa640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] conn(0x7f0540075600 0x7f0540077a40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.688+0000 7f609259c640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f608006be30 0x7f608006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.688+0000 7f7c457fa640 1 -- 172.21.15.77:0/3622683010 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7c40075e90 con 0x7f7c48165e90 2024-04-25T01:22:09.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.688+0000 7f6d0a7fc640 1 -- 172.21.15.77:0/3462736360 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6d10044070 con 0x7f6d14070a80 2024-04-25T01:22:09.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.688+0000 7f607b7fe640 1 -- 172.21.15.77:0/4246505091 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6088080540 con 0x7f609406ebc0 2024-04-25T01:22:09.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.688+0000 7f0560ff8640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f054006be80 0x7f054006e340 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f05540e6850 tx=0x7f054c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.688+0000 7f34f742a640 1 -- 172.21.15.77:0/1371674193 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f34f0068fb0 con 0x7f34f0147a80 2024-04-25T01:22:09.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f609259c640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f608006be30 0x7f608006e2f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f608400c2b0 tx=0x7f6084008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f39a2ffd640 1 -- 172.21.15.77:0/855530884 <== osd.5 v2:172.21.15.121:6817/3849358101 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39ac032e50 con 0x7f3994075680 2024-04-25T01:22:09.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.688+0000 7f7c47fff640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] conn(0x7f7c34075600 0x7f7c34077a40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f607b7fe640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] conn(0x7f60800754b0 0x7f60800778f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f607b7fe640 1 -- 172.21.15.77:0/4246505091 --> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6080077fc0 con 0x7f60800754b0 2024-04-25T01:22:09.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f6d0a7fc640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6cf806be80 0x7f6cf806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f6d0a7fc640 1 -- 172.21.15.77:0/3462736360 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6d1002ad20 con 0x7f6d14070a80 2024-04-25T01:22:09.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f6d19239640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6cf806be80 0x7f6cf806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f34e6ffd640 1 -- 172.21.15.77:0/1371674193 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f34e003d620 con 0x7f34f0147a80 2024-04-25T01:22:09.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f607b7fe640 1 -- 172.21.15.77:0/4246505091 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f60880b9050 con 0x7f609406ebc0 2024-04-25T01:22:09.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f1aadd9b640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ab00c6e30 0x7f1ab00ad1e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000c430 tx=0x7f1aa000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f6d0a7fc640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] conn(0x7f6cf8075600 0x7f6cf8077a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f6d0a7fc640 1 -- 172.21.15.77:0/3462736360 --> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d10001d30 con 0x7f6cf8075600 2024-04-25T01:22:09.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f6d0a7fc640 1 -- 172.21.15.77:0/3462736360 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6d10075fb0 con 0x7f6d14070a80 2024-04-25T01:22:09.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f0551ffb640 1 -- 172.21.15.77:0/828958658 <== osd.3 v2:172.21.15.77:6806/3656221607 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f055c06d100 con 0x7f0540075600 2024-04-25T01:22:09.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f609359e640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] conn(0x7f60800754b0 0x7f60800778f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f7c457fa640 1 -- 172.21.15.77:0/3622683010 <== osd.4 v2:172.21.15.121:6803/3650663429 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7c4003be50 con 0x7f7c34075600 2024-04-25T01:22:09.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f34e6ffd640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34d406c230 0x7f34d406e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.689+0000 7f1a9effd640 1 -- 172.21.15.77:0/3134658623 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa00074e0 con 0x7f1ab00c6e30 2024-04-25T01:22:09.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.690+0000 7f1a9effd640 1 -- 172.21.15.77:0/3134658623 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa00047e0 con 0x7f1ab00c6e30 2024-04-25T01:22:09.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.690+0000 7f1a9effd640 1 -- 172.21.15.77:0/3134658623 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa00074e0 con 0x7f1ab00c6e30 2024-04-25T01:22:09.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.690+0000 7f6d19239640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6cf806be80 0x7f6cf806e340 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6d1415fb10 tx=0x7f6d04008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.690+0000 7f6d1a23b640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] conn(0x7f6cf8075600 0x7f6cf8077a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.690+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/3134658623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab00add10 con 0x7f1ab00c6e30 2024-04-25T01:22:09.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.690+0000 7f0acf7fe640 1 -- 172.21.15.77:0/3464669368 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0adc045020 con 0x7f0ae80cec30 2024-04-25T01:22:09.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.690+0000 7f609359e640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] conn(0x7f60800754b0 0x7f60800778f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.690+0000 7f34f6428640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34d406c230 0x7f34d406e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.690+0000 7f0acf7fe640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ad406be80 0x7f0ad406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.690+0000 7f0acf7fe640 1 -- 172.21.15.77:0/3464669368 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0adc0ac360 con 0x7f0ae80cec30 2024-04-25T01:22:09.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f0acf7fe640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] conn(0x7f0ad4075600 0x7f0ad4077a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f0acf7fe640 1 -- 172.21.15.77:0/3464669368 --> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0adc007e60 con 0x7f0ad4075600 2024-04-25T01:22:09.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f0acf7fe640 1 -- 172.21.15.77:0/3464669368 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0adc0ac710 con 0x7f0ae80cec30 2024-04-25T01:22:09.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.690+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/3134658623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab00ae2d0 con 0x7f1ab00c6e30 2024-04-25T01:22:09.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f34f6428640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34d406c230 0x7f34d406e6f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009b20 tx=0x7f34ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f607b7fe640 1 -- 172.21.15.77:0/4246505091 <== osd.2 v2:172.21.15.77:6800/1130238186 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6080077fc0 con 0x7f60800754b0 2024-04-25T01:22:09.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/3134658623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1ab0068180 con 0x7f1ab00c6e30 2024-04-25T01:22:09.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f0ae6d9d640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ad406be80 0x7f0ad406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f0ae759e640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] conn(0x7f0ad4075600 0x7f0ad4077a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f6d1a23b640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] conn(0x7f6cf8075600 0x7f6cf8077a40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f34e6ffd640 1 -- 172.21.15.77:0/1371674193 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f34e003d420 con 0x7f34f0147a80 2024-04-25T01:22:09.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f34e6ffd640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x7f34d4076430 0x7f34d40788d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f34e6ffd640 1 -- 172.21.15.77:0/1371674193 --> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34d4078fa0 con 0x7f34d4076430 2024-04-25T01:22:09.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.691+0000 7f0ae6d9d640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ad406be80 0x7f0ad406e340 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8007fd0 tx=0x7f0ad8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.692+0000 7f34e6ffd640 1 -- 172.21.15.77:0/1371674193 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f34e0075e50 con 0x7f34f0147a80 2024-04-25T01:22:09.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.692+0000 7f0ae759e640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] conn(0x7f0ad4075600 0x7f0ad4077a40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.692+0000 7f6d0a7fc640 1 -- 172.21.15.77:0/3462736360 <== osd.6 v2:172.21.15.121:6816/3890411649 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d10001d30 con 0x7f6cf8075600 2024-04-25T01:22:09.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.692+0000 7f1a9effd640 1 -- 172.21.15.77:0/3134658623 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1aa0011070 con 0x7f1ab00c6e30 2024-04-25T01:22:09.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.693+0000 7f1a9effd640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1a9006c230 0x7f1a9006e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.693+0000 7f0acf7fe640 1 -- 172.21.15.77:0/3464669368 <== osd.1 v2:172.21.15.77:6808/3460510388 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0adc007e60 con 0x7f0ad4075600 2024-04-25T01:22:09.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.693+0000 7f1a9effd640 1 -- 172.21.15.77:0/3134658623 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1aa00072e0 con 0x7f1ab00c6e30 2024-04-25T01:22:09.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.693+0000 7f1aae59c640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1a9006c230 0x7f1a9006e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.694+0000 7f1a9effd640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] conn(0x7f1a90075830 0x7f1a90077c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.694+0000 7f1a9effd640 1 -- 172.21.15.77:0/3134658623 --> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a90078340 con 0x7f1a90075830 2024-04-25T01:22:09.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.694+0000 7f1a9effd640 1 -- 172.21.15.77:0/3134658623 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1aa008f060 con 0x7f1ab00c6e30 2024-04-25T01:22:09.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.692+0000 7f34f5c27640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x7f34d4076430 0x7f34d40788d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.693+0000 7f34f5c27640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x7f34d4076430 0x7f34d40788d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.694+0000 7f1aae59c640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1a9006c230 0x7f1a9006e6f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1aa403c000 tx=0x7f1aa4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.694+0000 7f1aaed9d640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] conn(0x7f1a90075830 0x7f1a90077c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.694+0000 7f34e6ffd640 1 -- 172.21.15.77:0/1371674193 <== osd.7 v2:172.21.15.121:6800/251765690 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f34d4078fa0 con 0x7f34d4076430 2024-04-25T01:22:09.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.695+0000 7f1aaed9d640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] conn(0x7f1a90075830 0x7f1a90077c70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.696+0000 7f1a9effd640 1 -- 172.21.15.77:0/3134658623 <== osd.0 v2:172.21.15.77:6824/4267986699 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1a90078340 con 0x7f1a90075830 2024-04-25T01:22:09.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.721+0000 7f7c4efe2640 1 -- 172.21.15.77:0/3622683010 --> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c480cd140 con 0x7f7c34075600 2024-04-25T01:22:09.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.721+0000 7f60797fa640 1 -- 172.21.15.77:0/4246505091 --> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f60940aaeb0 con 0x7f60800754b0 2024-04-25T01:22:09.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.721+0000 7f607b7fe640 1 -- 172.21.15.77:0/4246505091 <== osd.2 v2:172.21.15.77:6800/1130238186 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f60940aaeb0 con 0x7f60800754b0 2024-04-25T01:22:09.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.721+0000 7f7c457fa640 1 -- 172.21.15.77:0/3622683010 <== osd.4 v2:172.21.15.121:6803/3650663429 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7c480cd140 con 0x7f7c34075600 2024-04-25T01:22:09.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.722+0000 7f39b8d7e640 1 -- 172.21.15.77:0/855530884 --> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f39b4068180 con 0x7f3994075680 2024-04-25T01:22:09.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.722+0000 7f60990d4640 1 -- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] conn(0x7f60800754b0 msgr2=0x7f60800778f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.722+0000 7f7c2effd640 1 -- 172.21.15.77:0/3622683010 >> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] conn(0x7f7c34075600 msgr2=0x7f7c34077a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.722+0000 7f60990d4640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] conn(0x7f60800754b0 0x7f60800778f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.722+0000 7f7c2effd640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] conn(0x7f7c34075600 0x7f7c34077a40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.722+0000 7f7c2effd640 1 -- 172.21.15.77:0/3622683010 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c3406bf00 msgr2=0x7f7c3406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.722+0000 7f7c2effd640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c3406bf00 0x7f7c3406e3c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7c4806e7b0 tx=0x7f7c38008040 comp rx=0 tx=0).stop 2024-04-25T01:22:09.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f7c2effd640 1 -- 172.21.15.77:0/3622683010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48165e90 msgr2=0x7f7c480710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f7c2effd640 1 --2- 172.21.15.77:0/3622683010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48165e90 0x7f7c480710c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7c40030000 tx=0x7f7c40004a70 comp rx=0 tx=0).stop 2024-04-25T01:22:09.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f7c4d57f640 1 -- 172.21.15.77:0/3622683010 reap_dead start 2024-04-25T01:22:09.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f39a2ffd640 1 -- 172.21.15.77:0/855530884 <== osd.5 v2:172.21.15.121:6817/3849358101 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f39b4068180 con 0x7f3994075680 2024-04-25T01:22:09.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f60990d4640 1 -- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f608006be30 msgr2=0x7f608006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f60990d4640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f608006be30 0x7f608006e2f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f608400c2b0 tx=0x7f6084008040 comp rx=0 tx=0).stop 2024-04-25T01:22:09.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f60990d4640 1 -- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f609406ebc0 msgr2=0x7f60940add70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f60990d4640 1 --2- 172.21.15.77:0/4246505091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f609406ebc0 0x7f60940add70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6088000c00 tx=0x7f60880046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f609359e640 1 -- 172.21.15.77:0/4246505091 reap_dead start 2024-04-25T01:22:09.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f60990d4640 1 -- 172.21.15.77:0/4246505091 shutdown_connections 2024-04-25T01:22:09.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f60990d4640 1 -- 172.21.15.77:0/4246505091 >> 172.21.15.77:0/4246505091 conn(0x7f609406b300 msgr2=0x7f60941556d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f7c2effd640 1 -- 172.21.15.77:0/3622683010 shutdown_connections 2024-04-25T01:22:09.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f7c2effd640 1 -- 172.21.15.77:0/3622683010 >> 172.21.15.77:0/3622683010 conn(0x7f7c480b1660 msgr2=0x7f7c480723d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f39b8d7e640 1 -- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] conn(0x7f3994075680 msgr2=0x7f3994077ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.723+0000 7f39b8d7e640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] conn(0x7f3994075680 0x7f3994077ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f60990d4640 1 -- 172.21.15.77:0/4246505091 shutdown_connections 2024-04-25T01:22:09.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f60990d4640 1 -- 172.21.15.77:0/4246505091 wait complete. 2024-04-25T01:22:09.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f7c2effd640 1 -- 172.21.15.77:0/3622683010 shutdown_connections 2024-04-25T01:22:09.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f39b8d7e640 1 -- 172.21.15.77:0/855530884 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f399406bf00 msgr2=0x7f399406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f39b8d7e640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f399406bf00 0x7f399406e3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f39a40018b0 tx=0x7f39a4009040 comp rx=0 tx=0).stop 2024-04-25T01:22:09.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f39b8d7e640 1 -- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b40cfb60 msgr2=0x7f39b414e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f39b8d7e640 1 --2- 172.21.15.77:0/855530884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b40cfb60 0x7f39b414e3b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f39ac030000 tx=0x7f39ac004a40 comp rx=0 tx=0).stop 2024-04-25T01:22:09.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f7c2effd640 1 -- 172.21.15.77:0/3622683010 wait complete. 2024-04-25T01:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f39b2d9d640 1 -- 172.21.15.77:0/855530884 reap_dead start 2024-04-25T01:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f39b8d7e640 1 -- 172.21.15.77:0/855530884 shutdown_connections 2024-04-25T01:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f39b8d7e640 1 -- 172.21.15.77:0/855530884 >> 172.21.15.77:0/855530884 conn(0x7f39b40d3e50 msgr2=0x7f39b406cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f39b8d7e640 1 -- 172.21.15.77:0/855530884 shutdown_connections 2024-04-25T01:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f39b8d7e640 1 -- 172.21.15.77:0/855530884 wait complete. 2024-04-25T01:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/3462736360 --> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6d14072b60 con 0x7f6cf8075600 2024-04-25T01:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.724+0000 7f0563a5d640 1 -- 172.21.15.77:0/828958658 --> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f05540057e0 con 0x7f0540075600 2024-04-25T01:22:09.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.725+0000 7f6d0a7fc640 1 -- 172.21.15.77:0/3462736360 <== osd.6 v2:172.21.15.121:6816/3890411649 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6d14072b60 con 0x7f6cf8075600 2024-04-25T01:22:09.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.725+0000 7f0551ffb640 1 -- 172.21.15.77:0/828958658 <== osd.3 v2:172.21.15.77:6806/3656221607 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f05540057e0 con 0x7f0540075600 2024-04-25T01:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.725+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] conn(0x7f6cf8075600 msgr2=0x7f6cf8077a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.725+0000 7f6d1aa3c640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] conn(0x7f6cf8075600 0x7f6cf8077a40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f0563a5d640 1 -- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] conn(0x7f0540075600 msgr2=0x7f0540077a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f0563a5d640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] conn(0x7f0540075600 0x7f0540077a40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/3462736360 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6cf806be80 msgr2=0x7f6cf806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f6d1aa3c640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6cf806be80 0x7f6cf806e340 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6d1415fb10 tx=0x7f6d04008040 comp rx=0 tx=0).stop 2024-04-25T01:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/3462736360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d14070a80 msgr2=0x7f6d141342f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f6d1aa3c640 1 --2- 172.21.15.77:0/3462736360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d14070a80 0x7f6d141342f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6d10035040 tx=0x7f6d100046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f6d1a23b640 1 -- 172.21.15.77:0/3462736360 reap_dead start 2024-04-25T01:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f0aec808640 1 -- 172.21.15.77:0/3464669368 --> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0ae80b31a0 con 0x7f0ad4075600 2024-04-25T01:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f0563a5d640 1 -- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f054006be80 msgr2=0x7f054006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f0563a5d640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f054006be80 0x7f054006e340 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f05540e6850 tx=0x7f054c009040 comp rx=0 tx=0).stop 2024-04-25T01:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f0563a5d640 1 -- 172.21.15.77:0/828958658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f055403e840 msgr2=0x7f05540e1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f0563a5d640 1 --2- 172.21.15.77:0/828958658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f055403e840 0x7f05540e1b10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f055c06ac00 tx=0x7f055c06b050 comp rx=0 tx=0).stop 2024-04-25T01:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f0561ffa640 1 -- 172.21.15.77:0/828958658 reap_dead start 2024-04-25T01:22:09.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/3462736360 shutdown_connections 2024-04-25T01:22:09.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/3462736360 >> 172.21.15.77:0/3462736360 conn(0x7f6d1406b540 msgr2=0x7f6d14070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f0acf7fe640 1 -- 172.21.15.77:0/3464669368 <== osd.1 v2:172.21.15.77:6808/3460510388 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0ae80b31a0 con 0x7f0ad4075600 2024-04-25T01:22:09.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.726+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/3462736360 shutdown_connections 2024-04-25T01:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.727+0000 7f0563a5d640 1 -- 172.21.15.77:0/828958658 shutdown_connections 2024-04-25T01:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.727+0000 7f0563a5d640 1 -- 172.21.15.77:0/828958658 >> 172.21.15.77:0/828958658 conn(0x7f05540559e0 msgr2=0x7f05540380e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.727+0000 7f6d1aa3c640 1 -- 172.21.15.77:0/3462736360 wait complete. 2024-04-25T01:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.727+0000 7f0563a5d640 1 -- 172.21.15.77:0/828958658 shutdown_connections 2024-04-25T01:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.727+0000 7f0563a5d640 1 -- 172.21.15.77:0/828958658 wait complete. 2024-04-25T01:22:09.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.727+0000 7f0acd7fa640 1 -- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] conn(0x7f0ad4075600 msgr2=0x7f0ad4077a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.727+0000 7f0acd7fa640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] conn(0x7f0ad4075600 0x7f0ad4077a40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.728+0000 7f0acd7fa640 1 -- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ad406be80 msgr2=0x7f0ad406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.728+0000 7f34f742a640 1 -- 172.21.15.77:0/1371674193 --> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f34f0072b60 con 0x7f34d4076430 2024-04-25T01:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.728+0000 7f0acd7fa640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ad406be80 0x7f0ad406e340 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8007fd0 tx=0x7f0ad8009040 comp rx=0 tx=0).stop 2024-04-25T01:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.728+0000 7f0acd7fa640 1 -- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae80cec30 msgr2=0x7f0ae8156a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.728+0000 7f0acd7fa640 1 --2- 172.21.15.77:0/3464669368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae80cec30 0x7f0ae8156a20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0adc031000 tx=0x7f0adc0046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.728+0000 7f0ae759e640 1 -- 172.21.15.77:0/3464669368 reap_dead start 2024-04-25T01:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.728+0000 7f34e6ffd640 1 -- 172.21.15.77:0/1371674193 <== osd.7 v2:172.21.15.121:6800/251765690 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f34f0072b60 con 0x7f34d4076430 2024-04-25T01:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.729+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/3134658623 --> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1ab00b2810 con 0x7f1a90075830 2024-04-25T01:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.729+0000 7f0acd7fa640 1 -- 172.21.15.77:0/3464669368 shutdown_connections 2024-04-25T01:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.729+0000 7f0acd7fa640 1 -- 172.21.15.77:0/3464669368 >> 172.21.15.77:0/3464669368 conn(0x7f0ae806bad0 msgr2=0x7f0ae80b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.729+0000 7f1a9effd640 1 -- 172.21.15.77:0/3134658623 <== osd.0 v2:172.21.15.77:6824/4267986699 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1ab00b2810 con 0x7f1a90075830 2024-04-25T01:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.729+0000 7f34f742a640 1 -- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x7f34d4076430 msgr2=0x7f34d40788d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.729+0000 7f34f742a640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x7f34d4076430 0x7f34d40788d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.729+0000 7f34f742a640 1 -- 172.21.15.77:0/1371674193 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34d406c230 msgr2=0x7f34d406e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.729+0000 7f34f742a640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34d406c230 0x7f34d406e6f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009b20 tx=0x7f34ec008040 comp rx=0 tx=0).stop 2024-04-25T01:22:09.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.729+0000 7f34f742a640 1 -- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f0147a80 msgr2=0x7f34f0148a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.729+0000 7f34f742a640 1 --2- 172.21.15.77:0/1371674193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f0147a80 0x7f34f0148a90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f34e002a230 tx=0x7f34e00046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.729+0000 7f0acd7fa640 1 -- 172.21.15.77:0/3464669368 shutdown_connections 2024-04-25T01:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.730+0000 7f0acd7fa640 1 -- 172.21.15.77:0/3464669368 wait complete. 2024-04-25T01:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.730+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] conn(0x7f1a90075830 msgr2=0x7f1a90077c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.730+0000 7f1ab4e9e640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] conn(0x7f1a90075830 0x7f1a90077c70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.730+0000 7f34f6c29640 1 -- 172.21.15.77:0/1371674193 reap_dead start 2024-04-25T01:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.730+0000 7f34f742a640 1 -- 172.21.15.77:0/1371674193 shutdown_connections 2024-04-25T01:22:09.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.730+0000 7f34f742a640 1 -- 172.21.15.77:0/1371674193 >> 172.21.15.77:0/1371674193 conn(0x7f34f006b540 msgr2=0x7f34f0144a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.730+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1a9006c230 msgr2=0x7f1a9006e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.730+0000 7f1ab4e9e640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1a9006c230 0x7f1a9006e6f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1aa403c000 tx=0x7f1aa4008030 comp rx=0 tx=0).stop 2024-04-25T01:22:09.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.730+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ab00c6e30 msgr2=0x7f1ab00ad1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.730+0000 7f1ab4e9e640 1 --2- 172.21.15.77:0/3134658623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ab00c6e30 0x7f1ab00ad1e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000c430 tx=0x7f1aa000c8f0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.730+0000 7f1aaed9d640 1 -- 172.21.15.77:0/3134658623 reap_dead start 2024-04-25T01:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.731+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/3134658623 shutdown_connections 2024-04-25T01:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.731+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/3134658623 >> 172.21.15.77:0/3134658623 conn(0x7f1ab00d02b0 msgr2=0x7f1ab0066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.731+0000 7f34f742a640 1 -- 172.21.15.77:0/1371674193 shutdown_connections 2024-04-25T01:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.731+0000 7f34f742a640 1 -- 172.21.15.77:0/1371674193 wait complete. 2024-04-25T01:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.731+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/3134658623 shutdown_connections 2024-04-25T01:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.731+0000 7f1ab4e9e640 1 -- 172.21.15.77:0/3134658623 wait complete. 2024-04-25T01:22:09.766 INFO:teuthology.orchestra.run.smithi077.stdout:55834574858 2024-04-25T01:22:09.767 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T01:22:09.767 INFO:teuthology.orchestra.run.smithi077.stdout:55834574859 2024-04-25T01:22:09.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T01:22:09.768 INFO:teuthology.orchestra.run.smithi077.stdout:55834574860 2024-04-25T01:22:09.769 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T01:22:09.769 INFO:teuthology.orchestra.run.smithi077.stdout:55834574860 2024-04-25T01:22:09.770 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T01:22:09.770 INFO:teuthology.orchestra.run.smithi077.stdout:55834574859 2024-04-25T01:22:09.771 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T01:22:09.771 INFO:teuthology.orchestra.run.smithi077.stdout:55834574858 2024-04-25T01:22:09.772 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T01:22:09.772 INFO:teuthology.orchestra.run.smithi077.stdout:55834574860 2024-04-25T01:22:09.773 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T01:22:09.773 INFO:teuthology.orchestra.run.smithi077.stdout:55834574858 2024-04-25T01:22:09.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T01:22:09.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.991+0000 7fb5a9e2a640 1 Processor -- start 2024-04-25T01:22:09.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.991+0000 7fb5a9e2a640 1 -- start start 2024-04-25T01:22:09.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.991+0000 7fdfb96ce640 1 Processor -- start 2024-04-25T01:22:09.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.991+0000 7fdfb96ce640 1 -- start start 2024-04-25T01:22:09.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.991+0000 7fc1a39c8640 1 Processor -- start 2024-04-25T01:22:09.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.992+0000 7fb5a9e2a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb59c0e7740 0x7fb59c007430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.992+0000 7fc1a39c8640 1 -- start start 2024-04-25T01:22:09.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.992+0000 7f8012977640 1 Processor -- start 2024-04-25T01:22:09.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.992+0000 7fb5a9e2a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb59c0538d0 con 0x7fb59c008cc0 2024-04-25T01:22:09.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fb5a2ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb59c008cc0 0x7fb59c050220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56470/0 (socket says 172.21.15.77:56470) 2024-04-25T01:22:09.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fb5a37fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb59c0e7740 0x7fb59c007430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fb5a2ffd640 1 -- 172.21.15.77:0/1726595165 learned_addr learned my addr 172.21.15.77:0/1726595165 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:09.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/1726595165 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb59c053bb0 con 0x7fb59c009100 2024-04-25T01:22:09.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/1726595165 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb59c050930 con 0x7fb59c0e7740 2024-04-25T01:22:09.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fb5a27fc640 1 -- 172.21.15.77:0/1726595165 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 929470314 0 0) 0x7fb59c0538d0 con 0x7fb59c008cc0 2024-04-25T01:22:09.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fb5a27fc640 1 -- 172.21.15.77:0/1726595165 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb594003520 con 0x7fb59c008cc0 2024-04-25T01:22:09.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fc1a39c8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc194048100 0x7fc194006f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fc1a39c8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194054560 0x7fc19404f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fc1a39c8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc194005ec0 con 0x7fc194007460 2024-04-25T01:22:09.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fc1a39c8640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fc1940e7b90 con 0x7fc194048100 2024-04-25T01:22:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fc1a39c8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc194006a90 con 0x7fc194054560 2024-04-25T01:22:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fc1a0f63640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc194007460 0x7fc194053e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:59620/0 (socket says 172.21.15.77:59620) 2024-04-25T01:22:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fc1a0f63640 1 -- 172.21.15.77:0/1670910878 learned_addr learned my addr 172.21.15.77:0/1670910878 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fc1a1f65640 1 --2- 172.21.15.77:0/1670910878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194054560 0x7fc19404f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fb5a37fe640 1 -- 172.21.15.77:0/1726595165 >> v1:172.21.15.121:6789/0 conn(0x7fb59c009100 legacy=0x7fb59c0489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fb5a37fe640 1 -- 172.21.15.77:0/1726595165 >> v1:172.21.15.77:6789/0 conn(0x7fb59c008cc0 legacy=0x7fb59c050220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7fb5a37fe640 1 -- 172.21.15.77:0/1726595165 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59c053e90 con 0x7fb59c0e7740 2024-04-25T01:22:09.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fc1a1764640 1 --2- 172.21.15.77:0/1670910878 >> v2:172.21.15.77:3300/0 conn(0x7fc194048100 0x7fc194006f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.993+0000 7f8012977640 1 -- start start 2024-04-25T01:22:09.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7f8012977640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f800c0c7500 0x7f800c1626f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7f8012977640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f800c165d30 con 0x7f800c06b120 2024-04-25T01:22:09.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7f8012977640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f800c163000 con 0x7f800c1664f0 2024-04-25T01:22:09.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7f8012977640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f800c1632e0 con 0x7f800c0c7500 2024-04-25T01:22:09.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7f37fa8e0640 1 Processor -- start 2024-04-25T01:22:09.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fc193fff640 1 -- 172.21.15.77:0/1670910878 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1996847724 0 0) 0x7fc194005ec0 con 0x7fc194007460 2024-04-25T01:22:09.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fc193fff640 1 -- 172.21.15.77:0/1670910878 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc188003520 con 0x7fc194007460 2024-04-25T01:22:09.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7f37fa8e0640 1 -- start start 2024-04-25T01:22:09.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fc1a1f65640 1 -- 172.21.15.77:0/1670910878 >> v2:172.21.15.77:3300/0 conn(0x7fc194048100 msgr2=0x7fc194006f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fc1a1f65640 1 --2- 172.21.15.77:0/1670910878 >> v2:172.21.15.77:3300/0 conn(0x7fc194048100 0x7fc194006f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fc1a1f65640 1 -- 172.21.15.77:0/1670910878 >> v1:172.21.15.121:6789/0 conn(0x7fc194007460 legacy=0x7fc194053e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fdfb96ce640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb41669e0 0x7fdfb406c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fdfb96ce640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb4166e70 con 0x7fdfb40c3920 2024-04-25T01:22:09.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fdfb96ce640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb40ccca0 con 0x7fdfb406eec0 2024-04-25T01:22:09.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fdfb96ce640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb406f2a0 con 0x7fdfb41669e0 2024-04-25T01:22:09.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fc1a1f65640 1 -- 172.21.15.77:0/1670910878 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc194053280 con 0x7fc194054560 2024-04-25T01:22:09.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7f800bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f800c06b120 0x7f800c06b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56472/0 (socket says 172.21.15.77:56472) 2024-04-25T01:22:09.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7f800bfff640 1 -- 172.21.15.77:0/2024953902 learned_addr learned my addr 172.21.15.77:0/2024953902 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:09.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fc1a1764640 1 --2- 172.21.15.77:0/1670910878 >> v2:172.21.15.77:3300/0 conn(0x7fc194048100 0x7fc194006f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:09.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f800b7fe640 1 --2- 172.21.15.77:0/2024953902 >> v2:172.21.15.121:3300/0 conn(0x7f800c0c7500 0x7f800c1626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fdfb3fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fdfb406eec0 0x7fdfb40c3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:59642/0 (socket says 172.21.15.77:59642) 2024-04-25T01:22:09.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fdfb3fff640 1 -- 172.21.15.77:0/2921594860 learned_addr learned my addr 172.21.15.77:0/2921594860 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:09.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fdfb8ecd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdfb40c3920 0x7fdfb406d4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56486/0 (socket says 172.21.15.77:56486) 2024-04-25T01:22:09.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.994+0000 7fdfb37fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb41669e0 0x7fdfb406c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f800affd640 1 -- 172.21.15.77:0/2024953902 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762299166 0 0) 0x7f800c163000 con 0x7f800c1664f0 2024-04-25T01:22:09.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f800affd640 1 -- 172.21.15.77:0/2024953902 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ff4003520 con 0x7f800c1664f0 2024-04-25T01:22:09.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f800b7fe640 1 -- 172.21.15.77:0/2024953902 >> v1:172.21.15.121:6789/0 conn(0x7f800c1664f0 legacy=0x7f800c06dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f800b7fe640 1 -- 172.21.15.77:0/2024953902 >> v1:172.21.15.77:6789/0 conn(0x7f800c06b120 legacy=0x7f800c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f800b7fe640 1 -- 172.21.15.77:0/2024953902 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800c1635c0 con 0x7f800c0c7500 2024-04-25T01:22:09.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7fdfb2ffd640 1 -- 172.21.15.77:0/2921594860 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3967226120 0 0) 0x7fdfb40ccca0 con 0x7fdfb406eec0 2024-04-25T01:22:09.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7fdfb37fe640 1 -- 172.21.15.77:0/2921594860 >> v1:172.21.15.121:6789/0 conn(0x7fdfb406eec0 legacy=0x7fdfb40c3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7fdfb37fe640 1 -- 172.21.15.77:0/2921594860 >> v1:172.21.15.77:6789/0 conn(0x7fdfb40c3920 legacy=0x7fdfb406d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7fdfb37fe640 1 -- 172.21.15.77:0/2921594860 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb40ccf80 con 0x7fdfb41669e0 2024-04-25T01:22:09.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f30da11d640 1 Processor -- start 2024-04-25T01:22:09.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f30da11d640 1 -- start start 2024-04-25T01:22:09.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f37fa8e0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f37ec034020 0x7f37ec04fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7fb5a37fe640 1 --2- 172.21.15.77:0/1726595165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb59c0e7740 0x7fb59c007430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb598009720 tx=0x7fb59802a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=326e9e272ee8634d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f37fa8e0640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37ec054c10 con 0x7f37ec006a50 2024-04-25T01:22:09.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f37fa8e0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37ec053a50 con 0x7f37ec04b3c0 2024-04-25T01:22:09.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.995+0000 7f37fa8e0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f37ec0062e0 con 0x7f37ec034020 2024-04-25T01:22:09.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f37f98de640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f37ec04b3c0 0x7f37ec04f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:59656/0 (socket says 172.21.15.77:59656) 2024-04-25T01:22:09.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f37f98de640 1 -- 172.21.15.77:0/2867129187 learned_addr learned my addr 172.21.15.77:0/2867129187 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:09.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f37f90dd640 1 --2- 172.21.15.77:0/2867129187 >> v2:172.21.15.77:3300/0 conn(0x7f37ec034020 0x7f37ec04fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7fc1a1f65640 1 --2- 172.21.15.77:0/1670910878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194054560 0x7fc19404f930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc19c000b60 tx=0x7fc19c0a5e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6020351e447f5f84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7fb5a27fc640 1 -- 172.21.15.77:0/1726595165 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598037070 con 0x7fb59c0e7740 2024-04-25T01:22:09.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7fb5a27fc640 1 -- 172.21.15.77:0/1726595165 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59802ad70 con 0x7fb59c0e7740 2024-04-25T01:22:09.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f37f88dc640 1 -- 172.21.15.77:0/2867129187 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2561849899 0 0) 0x7f37ec053a50 con 0x7f37ec04b3c0 2024-04-25T01:22:09.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f37f88dc640 1 -- 172.21.15.77:0/2867129187 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37e0003520 con 0x7f37ec04b3c0 2024-04-25T01:22:09.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f800b7fe640 1 --2- 172.21.15.77:0/2024953902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800c0c7500 0x7f800c1626f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8004009dc0 tx=0x7f800402a720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6102b527ed7d3fab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:09.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f37f90dd640 1 -- 172.21.15.77:0/2867129187 >> v1:172.21.15.121:6789/0 conn(0x7f37ec04b3c0 legacy=0x7f37ec04f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7fb5a27fc640 1 -- 172.21.15.77:0/1726595165 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598032660 con 0x7fb59c0e7740 2024-04-25T01:22:09.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7fc193fff640 1 -- 172.21.15.77:0/1670910878 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c0b3070 con 0x7fc194054560 2024-04-25T01:22:09.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7fc193fff640 1 -- 172.21.15.77:0/1670910878 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc19c0a6be0 con 0x7fc194054560 2024-04-25T01:22:09.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7fc193fff640 1 -- 172.21.15.77:0/1670910878 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c0ae660 con 0x7fc194054560 2024-04-25T01:22:09.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f800affd640 1 -- 172.21.15.77:0/2024953902 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8004007740 con 0x7f800c0c7500 2024-04-25T01:22:09.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f800affd640 1 -- 172.21.15.77:0/2024953902 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80040078e0 con 0x7f800c0c7500 2024-04-25T01:22:09.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f30da11d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30cc03a230 0x7f30cc0e4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f30da11d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f30cc050dd0 0x7f30cc0e4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:09.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f30da11d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30cc04f5e0 con 0x7f30cc03a610 2024-04-25T01:22:09.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30da11d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f30cc0501f0 con 0x7f30cc03a230 2024-04-25T01:22:09.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30da11d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f30cc04f8c0 con 0x7f30cc050dd0 2024-04-25T01:22:09.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d3fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f30cc03a610 0x7f30cc0de710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56508/0 (socket says 172.21.15.77:56508) 2024-04-25T01:22:09.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d3fff640 1 -- 172.21.15.77:0/1937730639 learned_addr learned my addr 172.21.15.77:0/1937730639 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:09.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f800affd640 1 -- 172.21.15.77:0/2024953902 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8004007cb0 con 0x7f800c0c7500 2024-04-25T01:22:09.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/1726595165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb59c0e7740 msgr2=0x7fb59c007430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d2ffd640 1 --2- 172.21.15.77:0/1937730639 >> v2:172.21.15.121:3300/0 conn(0x7f30cc050dd0 0x7f30cc0e4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7fb5a9e2a640 1 --2- 172.21.15.77:0/1726595165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb59c0e7740 0x7fb59c007430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb598009720 tx=0x7fb59802a060 comp rx=0 tx=0).stop 2024-04-25T01:22:09.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/1726595165 shutdown_connections 2024-04-25T01:22:09.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7fb5a9e2a640 1 --2- 172.21.15.77:0/1726595165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb59c0e7740 0x7fb59c007430 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/1726595165 >> 172.21.15.77:0/1726595165 conn(0x7fb59c055bd0 msgr2=0x7fb59c055fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7fc1a39c8640 1 -- 172.21.15.77:0/1670910878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194054560 msgr2=0x7fc19404f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7fc1a39c8640 1 --2- 172.21.15.77:0/1670910878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194054560 0x7fc19404f930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc19c000b60 tx=0x7fc19c0a5e10 comp rx=0 tx=0).stop 2024-04-25T01:22:09.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f8568d29640 1 Processor -- start 2024-04-25T01:22:09.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f8568d29640 1 -- start start 2024-04-25T01:22:09.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d37fe640 1 --2- 172.21.15.77:0/1937730639 >> v2:172.21.15.77:3300/0 conn(0x7f30cc03a230 0x7f30cc0e4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:09.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2467242541 0 0) 0x7f30cc04f5e0 con 0x7f30cc03a610 2024-04-25T01:22:09.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30c4003520 con 0x7f30cc03a610 2024-04-25T01:22:09.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2893648453 0 0) 0x7f30c4003520 con 0x7f30cc03a610 2024-04-25T01:22:09.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 >> v2:172.21.15.121:3300/0 conn(0x7f30cc050dd0 msgr2=0x7f30cc0e4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7fc1a39c8640 1 -- 172.21.15.77:0/1670910878 shutdown_connections 2024-04-25T01:22:09.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7fc1a39c8640 1 --2- 172.21.15.77:0/1670910878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194054560 0x7fc19404f930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7fc1a39c8640 1 --2- 172.21.15.77:0/1670910878 >> v2:172.21.15.77:3300/0 conn(0x7fc194048100 0x7fc194006f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f07f2ab3640 1 Processor -- start 2024-04-25T01:22:09.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7fc1a39c8640 1 -- 172.21.15.77:0/1670910878 >> 172.21.15.77:0/1670910878 conn(0x7fc194009fe0 msgr2=0x7fc19400a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f37f90dd640 1 -- 172.21.15.77:0/2867129187 >> v1:172.21.15.77:6789/0 conn(0x7f37ec006a50 legacy=0x7f37ec006e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:09.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.996+0000 7f37f90dd640 1 -- 172.21.15.77:0/2867129187 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ec053d30 con 0x7f37ec034020 2024-04-25T01:22:09.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8012977640 1 -- 172.21.15.77:0/2024953902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800c0c7500 msgr2=0x7f800c1626f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:09.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8012977640 1 --2- 172.21.15.77:0/2024953902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800c0c7500 0x7f800c1626f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8004009dc0 tx=0x7f800402a720 comp rx=0 tx=0).stop 2024-04-25T01:22:09.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7fc1a39c8640 1 -- 172.21.15.77:0/1670910878 shutdown_connections 2024-04-25T01:22:09.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8012977640 1 -- 172.21.15.77:0/2024953902 shutdown_connections 2024-04-25T01:22:09.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8012977640 1 --2- 172.21.15.77:0/2024953902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800c0c7500 0x7f800c1626f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:09.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8012977640 1 -- 172.21.15.77:0/2024953902 >> 172.21.15.77:0/2024953902 conn(0x7f800c0b2670 msgr2=0x7f800c0b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:09.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7fc1a39c8640 1 -- 172.21.15.77:0/1670910878 wait complete. 2024-04-25T01:22:09.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/1726595165 shutdown_connections 2024-04-25T01:22:10.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8568d29640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f856406ee50 0x7f856406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8012977640 1 -- 172.21.15.77:0/2024953902 shutdown_connections 2024-04-25T01:22:10.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d27fc640 1 --2- 172.21.15.77:0/1937730639 >> v2:172.21.15.121:3300/0 conn(0x7f30cc050dd0 0x7f30cc0e4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 >> v2:172.21.15.77:3300/0 conn(0x7f30cc03a230 msgr2=0x7f30cc0e4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d27fc640 1 --2- 172.21.15.77:0/1937730639 >> v2:172.21.15.77:3300/0 conn(0x7f30cc03a230 0x7f30cc0e4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30cc04fba0 con 0x7f30cc03a610 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.997+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f30c0001670 con 0x7f30cc03a610 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 >> v1:172.21.15.77:6789/0 conn(0x7f30cc03a610 legacy=0x7f30cc0de710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f30d27fc640 1 --2- 172.21.15.77:0/1937730639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c4003a00 0x7f30c4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f30d27fc640 1 --2- 172.21.15.77:0/1937730639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c4004800 0x7f30c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f30d27fc640 1 --2- 172.21.15.77:0/1937730639 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30c4005210 0x7f30c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc0501f0 con 0x7f30c4003a00 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc04f8c0 con 0x7f30c4004800 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f30c400c1e0 con 0x7f30c4005210 2024-04-25T01:22:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f30d37fe640 1 --2- 172.21.15.77:0/1937730639 >> v2:172.21.15.77:3300/0 conn(0x7f30cc03a230 0x7f30cc0e4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f30d2ffd640 1 --2- 172.21.15.77:0/1937730639 >> v2:172.21.15.121:3300/0 conn(0x7f30cc050dd0 0x7f30cc0e4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f07f2ab3640 1 -- start start 2024-04-25T01:22:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8012977640 1 -- 172.21.15.77:0/2024953902 wait complete. 2024-04-25T01:22:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7fdfb37fe640 1 --2- 172.21.15.77:0/2921594860 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb41669e0 0x7fdfb406c7a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009dc0 tx=0x7fdfac02a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a33a9195277008b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7fdfb2ffd640 1 -- 172.21.15.77:0/2921594860 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac007740 con 0x7fdfb41669e0 2024-04-25T01:22:10.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7fdfb2ffd640 1 -- 172.21.15.77:0/2921594860 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfac0078e0 con 0x7fdfb41669e0 2024-04-25T01:22:10.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8568d29640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f85640717c0 0x7f856416b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8568d29640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85640b3710 con 0x7f856406d1d0 2024-04-25T01:22:10.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8568d29640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f85640b4040 con 0x7f85640717c0 2024-04-25T01:22:10.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8568d29640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f856406d610 con 0x7f856406ee50 2024-04-25T01:22:10.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8562d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f85640717c0 0x7f856416b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8562d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f85640717c0 0x7f856416b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56294/0 (socket says 172.21.15.77:56294) 2024-04-25T01:22:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f8562d9d640 1 -- 172.21.15.77:0/143879972 learned_addr learned my addr 172.21.15.77:0/143879972 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f856259c640 1 --2- 172.21.15.77:0/143879972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f856406ee50 0x7f856406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f07f2ab3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4048100 0x7f07e4006f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f07f2ab3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f07e4007460 0x7f07e4053e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f07f2ab3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f07e4054390 0x7f07e404f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f07f2ab3640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f07e40e7b90 con 0x7f07e4007460 2024-04-25T01:22:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f07f2ab3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f07e40547e0 con 0x7f07e4054390 2024-04-25T01:22:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f07f2ab3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f07e4006a90 con 0x7f07e4048100 2024-04-25T01:22:10.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7fdfb96ce640 1 -- 172.21.15.77:0/2921594860 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb41669e0 msgr2=0x7fdfb406c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7fdfb96ce640 1 --2- 172.21.15.77:0/2921594860 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb41669e0 0x7fdfb406c7a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009dc0 tx=0x7fdfac02a6d0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f37f90dd640 1 --2- 172.21.15.77:0/2867129187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ec034020 0x7f37ec04fbe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f37ec006460 tx=0x7f37f002a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acde2f0f2a5105d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f30d2ffd640 1 --2- 172.21.15.77:0/1937730639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c4004800 0x7f30c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7f30d37fe640 1 --2- 172.21.15.77:0/1937730639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c4003a00 0x7f30c4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f30d37fe640 1 -- 172.21.15.77:0/1937730639 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30c4005210 msgr2=0x7f30c400b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:22:10.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f30d37fe640 1 --2- 172.21.15.77:0/1937730639 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30c4005210 0x7f30c400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f30d37fe640 1 -- 172.21.15.77:0/1937730639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c4004800 msgr2=0x7f30c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:10.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f30d37fe640 1 --2- 172.21.15.77:0/1937730639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c4004800 0x7f30c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f30d37fe640 1 -- 172.21.15.77:0/1937730639 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c4003520 con 0x7f30c4003a00 2024-04-25T01:22:10.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f30d3fff640 1 -- 172.21.15.77:0/1937730639 reap_dead start 2024-04-25T01:22:10.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f07f084f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4048100 0x7f07e4006f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f07f084f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4048100 0x7f07e4006f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33794/0 (socket says 172.21.15.77:33794) 2024-04-25T01:22:10.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f07f084f640 1 -- 172.21.15.77:0/3613863535 learned_addr learned my addr 172.21.15.77:0/3613863535 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:10.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.998+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/1726595165 wait complete. 2024-04-25T01:22:10.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f37f88dc640 1 -- 172.21.15.77:0/2867129187 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0036070 con 0x7f37ec034020 2024-04-25T01:22:10.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f37f88dc640 1 -- 172.21.15.77:0/2867129187 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f0031600 con 0x7f37ec034020 2024-04-25T01:22:10.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:09.999+0000 7f07f1050640 1 --2- 172.21.15.77:0/3613863535 >> v2:172.21.15.121:3300/0 conn(0x7f07e4054390 0x7f07e404f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f07ebfff640 1 --2- 172.21.15.77:0/3613863535 >> v2:172.21.15.77:3300/0 conn(0x7f07e4007460 0x7f07e4053e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7fc1a39c8640 1 Processor -- start 2024-04-25T01:22:10.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f8012977640 1 Processor -- start 2024-04-25T01:22:10.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f856259c640 1 -- 172.21.15.77:0/143879972 >> v2:172.21.15.77:3300/0 conn(0x7f85640717c0 msgr2=0x7f856416b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:10.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f856159a640 1 -- 172.21.15.77:0/143879972 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 17043451 0 0) 0x7f85640b3710 con 0x7f856406d1d0 2024-04-25T01:22:10.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f07f084f640 1 -- 172.21.15.77:0/3613863535 >> v2:172.21.15.121:3300/0 conn(0x7f07e4054390 msgr2=0x7f07e404f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:10.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7fdfb96ce640 1 -- 172.21.15.77:0/2921594860 shutdown_connections 2024-04-25T01:22:10.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7fdfb96ce640 1 --2- 172.21.15.77:0/2921594860 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb41669e0 0x7fdfb406c7a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7fdfb96ce640 1 -- 172.21.15.77:0/2921594860 >> 172.21.15.77:0/2921594860 conn(0x7fdfb40727f0 msgr2=0x7fdfb4072c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f856259c640 1 --2- 172.21.15.77:0/143879972 >> v2:172.21.15.77:3300/0 conn(0x7f85640717c0 0x7f856416b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f856259c640 1 -- 172.21.15.77:0/143879972 >> v1:172.21.15.77:6789/0 conn(0x7f856406d1d0 legacy=0x7f85640710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f856259c640 1 -- 172.21.15.77:0/143879972 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85640b39f0 con 0x7f856406ee50 2024-04-25T01:22:10.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f8562d9d640 1 --2- 172.21.15.77:0/143879972 >> v2:172.21.15.77:3300/0 conn(0x7f85640717c0 0x7f856416b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:10.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7fb5a9e2a640 1 Processor -- start 2024-04-25T01:22:10.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7fdfb96ce640 1 -- 172.21.15.77:0/2921594860 shutdown_connections 2024-04-25T01:22:10.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f07f084f640 1 --2- 172.21.15.77:0/3613863535 >> v2:172.21.15.121:3300/0 conn(0x7f07e4054390 0x7f07e404f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f07f084f640 1 -- 172.21.15.77:0/3613863535 >> v2:172.21.15.77:3300/0 conn(0x7f07e4007460 msgr2=0x7f07e4053e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:10.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f30d37fe640 1 --2- 172.21.15.77:0/1937730639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c4003a00 0x7f30c4003e20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009c40 tx=0x7f30c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee551d30728e0820 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7fb5a9e2a640 1 -- start start 2024-04-25T01:22:10.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8009840 con 0x7f30c4003a00 2024-04-25T01:22:10.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c8004020 con 0x7f30c4003a00 2024-04-25T01:22:10.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f30d27fc640 1 -- 172.21.15.77:0/1937730639 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8004410 con 0x7f30c4003a00 2024-04-25T01:22:10.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f8012977640 1 -- start start 2024-04-25T01:22:10.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f37fa8e0640 1 -- 172.21.15.77:0/2867129187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ec034020 msgr2=0x7f37ec04fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7fdfb96ce640 1 -- 172.21.15.77:0/2921594860 wait complete. 2024-04-25T01:22:10.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7f37fa8e0640 1 --2- 172.21.15.77:0/2867129187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ec034020 0x7f37ec04fbe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f37ec006460 tx=0x7f37f002a440 comp rx=0 tx=0).stop 2024-04-25T01:22:10.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f07f084f640 1 --2- 172.21.15.77:0/3613863535 >> v2:172.21.15.77:3300/0 conn(0x7f07e4007460 0x7f07e4053e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f07f084f640 1 -- 172.21.15.77:0/3613863535 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e4005ec0 con 0x7f07e4048100 2024-04-25T01:22:10.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f07ebfff640 1 --2- 172.21.15.77:0/3613863535 >> v2:172.21.15.77:3300/0 conn(0x7f07e4007460 0x7f07e4053e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:10.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.000+0000 7fc1a39c8640 1 -- start start 2024-04-25T01:22:10.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7fc1a39c8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc194007460 0x7fc194185150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7fc1a39c8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194048100 0x7fc194179bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f856259c640 1 --2- 172.21.15.77:0/143879972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f856406ee50 0x7f856406cc90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f85580097a0 tx=0x7f855802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9769e12c72d74338 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7fc1a39c8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc194054560 0x7fc19417a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f856159a640 1 -- 172.21.15.77:0/143879972 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8558037070 con 0x7f856406ee50 2024-04-25T01:22:10.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f856159a640 1 -- 172.21.15.77:0/143879972 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8558032440 con 0x7f856406ee50 2024-04-25T01:22:10.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f8012977640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f800c06b120 0x7f800c0d0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f8012977640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f800c0c7500 0x7f800c14dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f8012977640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800c1664f0 0x7f800c0d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f8012977640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f800c160050 con 0x7f800c06b120 2024-04-25T01:22:10.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f8012977640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f800c162c30 con 0x7f800c1664f0 2024-04-25T01:22:10.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f8012977640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f800c15b520 con 0x7f800c0c7500 2024-04-25T01:22:10.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f30da11d640 1 -- 172.21.15.77:0/1937730639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c4003a00 msgr2=0x7f30c4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f30da11d640 1 --2- 172.21.15.77:0/1937730639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c4003a00 0x7f30c4003e20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009c40 tx=0x7f30c4005830 comp rx=0 tx=0).stop 2024-04-25T01:22:10.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f30da11d640 1 -- 172.21.15.77:0/1937730639 shutdown_connections 2024-04-25T01:22:10.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f30da11d640 1 --2- 172.21.15.77:0/1937730639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c4003a00 0x7f30c4003e20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f30da11d640 1 -- 172.21.15.77:0/1937730639 >> 172.21.15.77:0/1937730639 conn(0x7f30cc009fb0 msgr2=0x7f30cc00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7fb5a9e2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb59c008cc0 0x7fb59c051ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f8010f14640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800c1664f0 0x7f800c0d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f800bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f800c06b120 0x7f800c0d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f8010f14640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800c1664f0 0x7f800c0d0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48782/0 (socket says 172.21.15.77:48782) 2024-04-25T01:22:10.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f8010f14640 1 -- 172.21.15.77:0/465402458 learned_addr learned my addr 172.21.15.77:0/465402458 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:10.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f800bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f800c06b120 0x7f800c0d0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56322/0 (socket says 172.21.15.77:56322) 2024-04-25T01:22:10.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7fc1a0f63640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194048100 0x7fc194179bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7f856159a640 1 -- 172.21.15.77:0/143879972 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8558032810 con 0x7f856406ee50 2024-04-25T01:22:10.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fb5a9e2a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb59c009100 0x7fb59c052000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fb5a9e2a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb59c0e7740 0x7fb59c0e3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb59c008cc0 0x7fb59c051ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb59c008cc0 0x7fb59c051ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56328/0 (socket says 172.21.15.77:56328) 2024-04-25T01:22:10.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fb5a37fe640 1 -- 172.21.15.77:0/3470616456 learned_addr learned my addr 172.21.15.77:0/3470616456 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:10.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb59c0e7b20 con 0x7fb59c008cc0 2024-04-25T01:22:10.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb59c0095b0 con 0x7fb59c009100 2024-04-25T01:22:10.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb59c0e7ca0 con 0x7fb59c0e7740 2024-04-25T01:22:10.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fdfb96ce640 1 Processor -- start 2024-04-25T01:22:10.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f800b7fe640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f800c0c7500 0x7f800c14dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.001+0000 7fc1a0f63640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194048100 0x7fc194179bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33810/0 (socket says 172.21.15.77:33810) 2024-04-25T01:22:10.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8010f14640 1 -- 172.21.15.77:0/465402458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f800c0c7500 msgr2=0x7f800c14dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8010f14640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f800c0c7500 0x7f800c14dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8010f14640 1 -- 172.21.15.77:0/465402458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f800c06b120 msgr2=0x7f800c0d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8010f14640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f800c06b120 0x7f800c0d0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8010f14640 1 -- 172.21.15.77:0/465402458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800c14e1c0 con 0x7f800c1664f0 2024-04-25T01:22:10.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fb5a2ffd640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb59c009100 0x7fb59c052000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fb5a3fff640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb59c0e7740 0x7fb59c0e3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7fdfb96ce640 1 -- start start 2024-04-25T01:22:10.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f37fa8e0640 1 -- 172.21.15.77:0/2867129187 shutdown_connections 2024-04-25T01:22:10.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f37fa8e0640 1 --2- 172.21.15.77:0/2867129187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ec034020 0x7f37ec04fbe0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f37fa8e0640 1 -- 172.21.15.77:0/2867129187 >> 172.21.15.77:0/2867129187 conn(0x7f37ec00a400 msgr2=0x7f37ec00a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8568d29640 1 -- 172.21.15.77:0/143879972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f856406ee50 msgr2=0x7f856406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8568d29640 1 --2- 172.21.15.77:0/143879972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f856406ee50 0x7f856406cc90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f85580097a0 tx=0x7f855802a180 comp rx=0 tx=0).stop 2024-04-25T01:22:10.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f37fa8e0640 1 -- 172.21.15.77:0/2867129187 shutdown_connections 2024-04-25T01:22:10.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f37fa8e0640 1 -- 172.21.15.77:0/2867129187 wait complete. 2024-04-25T01:22:10.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8010f14640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800c1664f0 0x7f800c0d0c30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f800000c3a0 tx=0x7f800000c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f07f084f640 1 --2- 172.21.15.77:0/3613863535 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4048100 0x7f07e4006f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f07e40514a0 tx=0x7f07e002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a97fc7fc6248bf8b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f07eb7fe640 1 -- 172.21.15.77:0/3613863535 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e0037070 con 0x7f07e4048100 2024-04-25T01:22:10.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f30da11d640 1 -- 172.21.15.77:0/1937730639 shutdown_connections 2024-04-25T01:22:10.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f30da11d640 1 -- 172.21.15.77:0/1937730639 wait complete. 2024-04-25T01:22:10.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f800bfff640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f800c06b120 0x7f800c0d0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:10.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f07eb7fe640 1 -- 172.21.15.77:0/3613863535 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07e0032440 con 0x7f07e4048100 2024-04-25T01:22:10.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8008ff9640 1 -- 172.21.15.77:0/465402458 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8000017070 con 0x7f800c1664f0 2024-04-25T01:22:10.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8008ff9640 1 -- 172.21.15.77:0/465402458 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80000052d0 con 0x7f800c1664f0 2024-04-25T01:22:10.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8568d29640 1 -- 172.21.15.77:0/143879972 shutdown_connections 2024-04-25T01:22:10.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8568d29640 1 --2- 172.21.15.77:0/143879972 >> v2:172.21.15.77:3300/0 conn(0x7f85640717c0 0x7f856416b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8568d29640 1 --2- 172.21.15.77:0/143879972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f856406ee50 0x7f856406cc90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.002+0000 7f8568d29640 1 -- 172.21.15.77:0/143879972 >> 172.21.15.77:0/143879972 conn(0x7f85640b1760 msgr2=0x7f85640b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7f8012977640 1 -- 172.21.15.77:0/465402458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f800c0cf1b0 con 0x7f800c1664f0 2024-04-25T01:22:10.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fb5a37fe640 1 -- 172.21.15.77:0/3470616456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb59c0e7740 msgr2=0x7fb59c0e3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fb5a37fe640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb59c0e7740 0x7fb59c0e3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fb5a37fe640 1 -- 172.21.15.77:0/3470616456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb59c009100 msgr2=0x7fb59c052000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fb5a37fe640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb59c009100 0x7fb59c052000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fb5a37fe640 1 -- 172.21.15.77:0/3470616456 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59c0e43c0 con 0x7fb59c008cc0 2024-04-25T01:22:10.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fb5a3fff640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb59c0e7740 0x7fb59c0e3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:10.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fdfb96ce640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfb406eec0 0x7fdfb4099a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fc1a1f65640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc194054560 0x7fc19417a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fdfb96ce640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb40c3920 0x7fdfb4162ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fdfb96ce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb4163010 0x7fdfb4164b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fdfb96ce640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb40cda60 con 0x7fdfb4163010 2024-04-25T01:22:10.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fdfb96ce640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb40cd5d0 con 0x7fdfb406eec0 2024-04-25T01:22:10.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fdfb96ce640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb40cdbe0 con 0x7fdfb40c3920 2024-04-25T01:22:10.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fc1a1f65640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc194054560 0x7fc19417a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56340/0 (socket says 172.21.15.77:56340) 2024-04-25T01:22:10.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fdfb8ecd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb4163010 0x7fdfb4164b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7f8008ff9640 1 -- 172.21.15.77:0/465402458 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8000003810 con 0x7f800c1664f0 2024-04-25T01:22:10.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fdfb8ecd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb4163010 0x7fdfb4164b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56354/0 (socket says 172.21.15.77:56354) 2024-04-25T01:22:10.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fdfb8ecd640 1 -- 172.21.15.77:0/958613361 learned_addr learned my addr 172.21.15.77:0/958613361 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:10.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7f8012977640 1 -- 172.21.15.77:0/465402458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f800c0cf770 con 0x7f800c1664f0 2024-04-25T01:22:10.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fc1a0f63640 1 -- 172.21.15.77:0/1662845202 learned_addr learned my addr 172.21.15.77:0/1662845202 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:10.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7f07f2ab3640 1 -- 172.21.15.77:0/3613863535 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4048100 msgr2=0x7f07e4006f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7f07f2ab3640 1 --2- 172.21.15.77:0/3613863535 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4048100 0x7f07e4006f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f07e40514a0 tx=0x7f07e002a270 comp rx=0 tx=0).stop 2024-04-25T01:22:10.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fc1a39c8640 1 -- 172.21.15.77:0/1662845202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc194006720 con 0x7fc194054560 2024-04-25T01:22:10.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fc1a39c8640 1 -- 172.21.15.77:0/1662845202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc194006440 con 0x7fc194007460 2024-04-25T01:22:10.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7fc1a39c8640 1 -- 172.21.15.77:0/1662845202 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc19404fe70 con 0x7fc194048100 2024-04-25T01:22:10.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.003+0000 7f8568d29640 1 -- 172.21.15.77:0/143879972 shutdown_connections 2024-04-25T01:22:10.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7f07f2ab3640 1 -- 172.21.15.77:0/3613863535 shutdown_connections 2024-04-25T01:22:10.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7f07f2ab3640 1 --2- 172.21.15.77:0/3613863535 >> v2:172.21.15.121:3300/0 conn(0x7f07e4054390 0x7f07e404f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fdfb3fff640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfb406eec0 0x7fdfb4099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7f37fa8e0640 1 Processor -- start 2024-04-25T01:22:10.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7f37fa8e0640 1 -- start start 2024-04-25T01:22:10.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7f07f2ab3640 1 --2- 172.21.15.77:0/3613863535 >> v2:172.21.15.77:3300/0 conn(0x7f07e4007460 0x7f07e4053e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fb5a37fe640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb59c008cc0 0x7fb59c051ac0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb598035040 tx=0x7fb5980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7f07f2ab3640 1 --2- 172.21.15.77:0/3613863535 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4048100 0x7f07e4006f20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fdfb8ecd640 1 -- 172.21.15.77:0/958613361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb40c3920 msgr2=0x7fdfb4162ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fdfb8ecd640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb40c3920 0x7fdfb4162ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fdfb8ecd640 1 -- 172.21.15.77:0/958613361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfb406eec0 msgr2=0x7fdfb4099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fdfb8ecd640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfb406eec0 0x7fdfb4099a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fdfb8ecd640 1 -- 172.21.15.77:0/958613361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb41634a0 con 0x7fdfb4163010 2024-04-25T01:22:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7f07f2ab3640 1 -- 172.21.15.77:0/3613863535 >> 172.21.15.77:0/3613863535 conn(0x7f07e4009fe0 msgr2=0x7f07e400a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7f30da11d640 1 Processor -- start 2024-04-25T01:22:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fb58ffff640 1 -- 172.21.15.77:0/3470616456 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598041070 con 0x7fb59c008cc0 2024-04-25T01:22:10.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fdfb8ecd640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb4163010 0x7fdfb4164b70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400c3f0 tx=0x7fdfa400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fc1a1764640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc194007460 0x7fc194185150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7f30da11d640 1 -- start start 2024-04-25T01:22:10.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fdfb0ff9640 1 -- 172.21.15.77:0/958613361 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa40074e0 con 0x7fdfb4163010 2024-04-25T01:22:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7f07f2ab3640 1 -- 172.21.15.77:0/3613863535 shutdown_connections 2024-04-25T01:22:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7fdfb96ce640 1 -- 172.21.15.77:0/958613361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb41f8200 con 0x7fdfb4163010 2024-04-25T01:22:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fc1a1764640 1 -- 172.21.15.77:0/1662845202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194048100 msgr2=0x7fc194179bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7fc1a1764640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194048100 0x7fc194179bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7fc1a1764640 1 -- 172.21.15.77:0/1662845202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc194054560 msgr2=0x7fc19417a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7fc1a1764640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc194054560 0x7fc19417a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7fc1a1764640 1 -- 172.21.15.77:0/1662845202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19417a740 con 0x7fc194007460 2024-04-25T01:22:10.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f8012977640 1 -- 172.21.15.77:0/465402458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f800c068180 con 0x7f800c1664f0 2024-04-25T01:22:10.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f37fa8e0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37ec006a50 0x7f37ec0db2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f37fa8e0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37ec034020 0x7f37ec043ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f37fa8e0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ec04b3c0 0x7f37ec045530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f37f98de640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37ec006a50 0x7f37ec0db2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f37f98de640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37ec006a50 0x7f37ec0db2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33860/0 (socket says 172.21.15.77:33860) 2024-04-25T01:22:10.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f37f98de640 1 -- 172.21.15.77:0/60631021 learned_addr learned my addr 172.21.15.77:0/60631021 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec04b7a0 con 0x7f37ec04b3c0 2024-04-25T01:22:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec050120 con 0x7f37ec034020 2024-04-25T01:22:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f37ec054830 con 0x7f37ec006a50 2024-04-25T01:22:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f37fa0df640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ec04b3c0 0x7f37ec045530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f30da11d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c4003a00 0x7f30cc04db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f30da11d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30cc04ce40 0x7f30cc035690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f30da11d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30cc035bd0 0x7f30cc0d1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f30da11d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc038630 con 0x7f30c4003a00 2024-04-25T01:22:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f30da11d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc034130 con 0x7f30cc04ce40 2024-04-25T01:22:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f30da11d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f30cc0387b0 con 0x7f30cc035bd0 2024-04-25T01:22:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f30d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c4003a00 0x7f30cc04db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7fc1a1764640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc194007460 0x7fc194185150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00e900 tx=0x7fc18c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.004+0000 7f8568d29640 1 -- 172.21.15.77:0/143879972 wait complete. 2024-04-25T01:22:10.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f8568d29640 1 Processor -- start 2024-04-25T01:22:10.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f8008ff9640 1 -- 172.21.15.77:0/465402458 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f800001a070 con 0x7f800c1664f0 2024-04-25T01:22:10.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7fdfb0ff9640 1 -- 172.21.15.77:0/958613361 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa4007680 con 0x7fdfb4163010 2024-04-25T01:22:10.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7fdfb96ce640 1 -- 172.21.15.77:0/958613361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb4165b30 con 0x7fdfb4163010 2024-04-25T01:22:10.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7fdfb0ff9640 1 -- 172.21.15.77:0/958613361 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa40049e0 con 0x7fdfb4163010 2024-04-25T01:22:10.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f37fa0df640 1 -- 172.21.15.77:0/60631021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37ec006a50 msgr2=0x7f37ec0db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f37fa0df640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37ec006a50 0x7f37ec0db2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f37fa0df640 1 -- 172.21.15.77:0/60631021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37ec034020 msgr2=0x7f37ec043ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:22:10.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f37fa0df640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37ec034020 0x7f37ec043ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f37fa0df640 1 -- 172.21.15.77:0/60631021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ec044490 con 0x7f37ec04b3c0 2024-04-25T01:22:10.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f37fa0df640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ec04b3c0 0x7f37ec045530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f37e400c410 tx=0x7f37e400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7f07f2ab3640 1 -- 172.21.15.77:0/3613863535 wait complete. 2024-04-25T01:22:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f07f2ab3640 1 Processor -- start 2024-04-25T01:22:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f8008ff9640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7ff806be30 0x7f7ff806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f8008ff9640 1 -- 172.21.15.77:0/465402458 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f800008a5e0 con 0x7f800c1664f0 2024-04-25T01:22:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f8568d29640 1 -- start start 2024-04-25T01:22:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f30d2ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30cc04ce40 0x7f30cc035690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f30d2ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30cc04ce40 0x7f30cc035690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48818/0 (socket says 172.21.15.77:48818) 2024-04-25T01:22:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f30d2ffd640 1 -- 172.21.15.77:0/1134961771 learned_addr learned my addr 172.21.15.77:0/1134961771 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f37ea7fc640 1 -- 172.21.15.77:0/60631021 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4017070 con 0x7f37ec04b3c0 2024-04-25T01:22:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37ec044770 con 0x7f37ec04b3c0 2024-04-25T01:22:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f8568d29640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f856406d1d0 0x7f85642039a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f8568d29640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f856406ee50 0x7f8564164210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f8568d29640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85640717c0 0x7f856416ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f8568d29640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f856406f3b0 con 0x7f856406ee50 2024-04-25T01:22:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f8568d29640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f856406f230 con 0x7f856406d1d0 2024-04-25T01:22:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f8568d29640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f856408b8d0 con 0x7f85640717c0 2024-04-25T01:22:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37ec044d30 con 0x7f37ec04b3c0 2024-04-25T01:22:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7fdfb96ce640 1 -- 172.21.15.77:0/958613361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb4068180 con 0x7fdfb4163010 2024-04-25T01:22:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f800bfff640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7ff806be30 0x7f7ff806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7fb58ffff640 1 -- 172.21.15.77:0/3470616456 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59803c460 con 0x7fb59c008cc0 2024-04-25T01:22:10.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.005+0000 7fb58ffff640 1 -- 172.21.15.77:0/3470616456 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598037040 con 0x7fb59c008cc0 2024-04-25T01:22:10.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb59c0e3e00 con 0x7fb59c008cc0 2024-04-25T01:22:10.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb59c0dc7b0 con 0x7fb59c008cc0 2024-04-25T01:22:10.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f37ea7fc640 1 -- 172.21.15.77:0/60631021 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37e40035f0 con 0x7f37ec04b3c0 2024-04-25T01:22:10.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f30d3fff640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30cc035bd0 0x7f30cc0d1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7fc191ffb640 1 -- 172.21.15.77:0/1662845202 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c017070 con 0x7fc194007460 2024-04-25T01:22:10.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7fc191ffb640 1 -- 172.21.15.77:0/1662845202 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc18c00ca00 con 0x7fc194007460 2024-04-25T01:22:10.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7fc1a39c8640 1 -- 172.21.15.77:0/1662845202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc19417aa20 con 0x7fc194007460 2024-04-25T01:22:10.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7fc191ffb640 1 -- 172.21.15.77:0/1662845202 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c00cdd0 con 0x7fc194007460 2024-04-25T01:22:10.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f856259c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f856406d1d0 0x7f85642039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f856259c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f856406d1d0 0x7f85642039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48828/0 (socket says 172.21.15.77:48828) 2024-04-25T01:22:10.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7f856259c640 1 -- 172.21.15.77:0/1669890107 learned_addr learned my addr 172.21.15.77:0/1669890107 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:10.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.006+0000 7f07f2ab3640 1 -- start start 2024-04-25T01:22:10.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f07f2ab3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07e4007460 0x7f07e40e9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f07f2ab3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07e4048100 0x7f07e40eb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f07f2ab3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4054390 0x7f07e40eb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f07f2ab3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f07e4006440 con 0x7f07e4007460 2024-04-25T01:22:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f07f2ab3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f07e4006160 con 0x7f07e4048100 2024-04-25T01:22:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f07f2ab3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f07e4006720 con 0x7f07e4054390 2024-04-25T01:22:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.007+0000 7fdfb0ff9640 1 -- 172.21.15.77:0/958613361 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdfa401a020 con 0x7fdfb4163010 2024-04-25T01:22:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7fdfb0ff9640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdfa006bf30 0x7fdfa006e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f37ea7fc640 1 -- 172.21.15.77:0/60631021 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e40039e0 con 0x7f37ec04b3c0 2024-04-25T01:22:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37dc005290 con 0x7f37ec04b3c0 2024-04-25T01:22:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f07ebfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07e4048100 0x7f07e40eb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f07ebfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07e4048100 0x7f07e40eb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48838/0 (socket says 172.21.15.77:48838) 2024-04-25T01:22:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f07ebfff640 1 -- 172.21.15.77:0/863825972 learned_addr learned my addr 172.21.15.77:0/863825972 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f07f1050640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4054390 0x7f07e40eb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f856259c640 1 -- 172.21.15.77:0/1669890107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85640717c0 msgr2=0x7f856416ca40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:22:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f856259c640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85640717c0 0x7f856416ca40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f856259c640 1 -- 172.21.15.77:0/1669890107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f856406ee50 msgr2=0x7f8564164210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f856259c640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f856406ee50 0x7f8564164210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7f856259c640 1 -- 172.21.15.77:0/1669890107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85641624e0 con 0x7f856406d1d0 2024-04-25T01:22:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7fdfb0ff9640 1 -- 172.21.15.77:0/958613361 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdfa408a8c0 con 0x7fdfb4163010 2024-04-25T01:22:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f800bfff640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7ff806be30 0x7f7ff806e2f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc005880 tx=0x7f7ffc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7fdfb3fff640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdfa006bf30 0x7fdfa006e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f07f1050640 1 -- 172.21.15.77:0/863825972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07e4048100 msgr2=0x7f07e40eb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f07f1050640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07e4048100 0x7f07e40eb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f07f1050640 1 -- 172.21.15.77:0/863825972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07e4007460 msgr2=0x7f07e40e9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f07f1050640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07e4007460 0x7f07e40e9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f07f1050640 1 -- 172.21.15.77:0/863825972 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e40cef20 con 0x7f07e4054390 2024-04-25T01:22:10.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.008+0000 7fc1a39c8640 1 -- 172.21.15.77:0/1662845202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1940e3db0 con 0x7fc194007460 2024-04-25T01:22:10.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f07ebfff640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07e4048100 0x7f07e40eb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:10.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f30d2ffd640 1 -- 172.21.15.77:0/1134961771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30cc035bd0 msgr2=0x7f30cc0d1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f30d2ffd640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30cc035bd0 0x7f30cc0d1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f30d2ffd640 1 -- 172.21.15.77:0/1134961771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c4003a00 msgr2=0x7f30cc04db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f30d2ffd640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c4003a00 0x7f30cc04db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f30d2ffd640 1 -- 172.21.15.77:0/1134961771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30cc04bda0 con 0x7f30cc04ce40 2024-04-25T01:22:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb59c0057e0 con 0x7fb59c008cc0 2024-04-25T01:22:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7fdfb3fff640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdfa006bf30 0x7fdfa006e3f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8000f60 tx=0x7fdfa8007b90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f07f1050640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4054390 0x7f07e40eb7a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f07d800d550 tx=0x7f07d800d950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7fc1877fe640 1 -- 172.21.15.77:0/1662845202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc19400ad00 con 0x7fc194007460 2024-04-25T01:22:10.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f856259c640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f856406d1d0 0x7f85642039a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f855802a770 tx=0x7f85580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f30d2ffd640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30cc04ce40 0x7f30cc035690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f30d4064f60 tx=0x7f30d4073800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f07e97fa640 1 -- 172.21.15.77:0/863825972 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8018020 con 0x7f07e4054390 2024-04-25T01:22:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f07e97fa640 1 -- 172.21.15.77:0/863825972 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d8014620 con 0x7f07e4054390 2024-04-25T01:22:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.009+0000 7f07f2ab3640 1 -- 172.21.15.77:0/863825972 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e40cf200 con 0x7f07e4054390 2024-04-25T01:22:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.010+0000 7f07e97fa640 1 -- 172.21.15.77:0/863825972 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8014a10 con 0x7f07e4054390 2024-04-25T01:22:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.010+0000 7f07f2ab3640 1 -- 172.21.15.77:0/863825972 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e40cf7c0 con 0x7f07e4054390 2024-04-25T01:22:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.010+0000 7f07f2ab3640 1 -- 172.21.15.77:0/863825972 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07e40053c0 con 0x7f07e4054390 2024-04-25T01:22:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.011+0000 7f07e97fa640 1 -- 172.21.15.77:0/863825972 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f07d801b020 con 0x7f07e4054390 2024-04-25T01:22:10.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.011+0000 7f07e97fa640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f07d006be30 0x7f07d006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f30bffff640 1 -- 172.21.15.77:0/1134961771 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d40503f0 con 0x7f30cc04ce40 2024-04-25T01:22:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f8008ff9640 1 -- 172.21.15.77:0/465402458 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f80000584d0 con 0x7f800c1664f0 2024-04-25T01:22:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f07e97fa640 1 -- 172.21.15.77:0/863825972 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f07d808b470 con 0x7f07e4054390 2024-04-25T01:22:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f07f084f640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f07d006be30 0x7f07d006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f30bffff640 1 -- 172.21.15.77:0/1134961771 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d4073e30 con 0x7f30cc04ce40 2024-04-25T01:22:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f30bffff640 1 -- 172.21.15.77:0/1134961771 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4074200 con 0x7f30cc04ce40 2024-04-25T01:22:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f8552ffd640 1 -- 172.21.15.77:0/1669890107 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8558037050 con 0x7f856406d1d0 2024-04-25T01:22:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f8568d29640 1 -- 172.21.15.77:0/1669890107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85641627c0 con 0x7f856406d1d0 2024-04-25T01:22:10.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f8568d29640 1 -- 172.21.15.77:0/1669890107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8564162d80 con 0x7f856406d1d0 2024-04-25T01:22:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f30da11d640 1 -- 172.21.15.77:0/1134961771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30cc04c080 con 0x7f30cc04ce40 2024-04-25T01:22:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f30da11d640 1 -- 172.21.15.77:0/1134961771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30cc04c640 con 0x7f30cc04ce40 2024-04-25T01:22:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f8552ffd640 1 -- 172.21.15.77:0/1669890107 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8558038040 con 0x7f856406d1d0 2024-04-25T01:22:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f8552ffd640 1 -- 172.21.15.77:0/1669890107 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855803ccf0 con 0x7f856406d1d0 2024-04-25T01:22:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.012+0000 7f07f084f640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f07d006be30 0x7f07d006e2f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f07e0034000 tx=0x7f07e002c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.013+0000 7f30da11d640 1 -- 172.21.15.77:0/1134961771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30cc0e5b90 con 0x7f30cc04ce40 2024-04-25T01:22:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.013+0000 7f8568d29640 1 -- 172.21.15.77:0/1669890107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8548005290 con 0x7f856406d1d0 2024-04-25T01:22:10.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.013+0000 7fc191ffb640 1 -- 172.21.15.77:0/1662845202 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc18c01a070 con 0x7fc194007460 2024-04-25T01:22:10.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.014+0000 7fc191ffb640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc18006bf00 0x7fc18006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.014+0000 7fc1a0f63640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc18006bf00 0x7fc18006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.014+0000 7fc191ffb640 1 -- 172.21.15.77:0/1662845202 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc18c08b780 con 0x7fc194007460 2024-04-25T01:22:10.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.015+0000 7fc1a0f63640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc18006bf00 0x7fc18006e3c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc1940e6d30 tx=0x7fc198009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.015+0000 7fdfb0ff9640 1 -- 172.21.15.77:0/958613361 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdfb41f8200 con 0x7fdfb4163010 2024-04-25T01:22:10.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.016+0000 7f07e97fa640 1 -- 172.21.15.77:0/863825972 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f07d8022030 con 0x7f07e4054390 2024-04-25T01:22:10.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.016+0000 7f37ea7fc640 1 -- 172.21.15.77:0/60631021 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f37e401a070 con 0x7f37ec04b3c0 2024-04-25T01:22:10.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.016+0000 7fb58ffff640 1 -- 172.21.15.77:0/3470616456 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb598044070 con 0x7fb59c008cc0 2024-04-25T01:22:10.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.017+0000 7f37ea7fc640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f37d806beb0 0x7f37d806e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.017+0000 7fb58ffff640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb58806be80 0x7fb58806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.017+0000 7fc191ffb640 1 -- 172.21.15.77:0/1662845202 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc18c059810 con 0x7fc194007460 2024-04-25T01:22:10.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.017+0000 7f37f98de640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f37d806beb0 0x7f37d806e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.017+0000 7fb5a2ffd640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb58806be80 0x7fb58806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.017+0000 7fb58ffff640 1 -- 172.21.15.77:0/3470616456 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb5980aba20 con 0x7fb59c008cc0 2024-04-25T01:22:10.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.018+0000 7fb5a2ffd640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb58806be80 0x7fb58806e340 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb59c04bf00 tx=0x7fb590007b80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.018+0000 7f37f98de640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f37d806beb0 0x7f37d806e370 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f37f4062d00 tx=0x7f37f4067eb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.018+0000 7f37ea7fc640 1 -- 172.21.15.77:0/60631021 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f37e408a960 con 0x7f37ec04b3c0 2024-04-25T01:22:10.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.019+0000 7f8552ffd640 1 -- 172.21.15.77:0/1669890107 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f85580503f0 con 0x7f856406d1d0 2024-04-25T01:22:10.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.019+0000 7f8552ffd640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f854406be30 0x7f854406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.019+0000 7f8561d9b640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f854406be30 0x7f854406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.019+0000 7f30bffff640 1 -- 172.21.15.77:0/1134961771 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f30d4074540 con 0x7f30cc04ce40 2024-04-25T01:22:10.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.020+0000 7f8552ffd640 1 -- 172.21.15.77:0/1669890107 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f85580afbc0 con 0x7f856406d1d0 2024-04-25T01:22:10.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.020+0000 7f30bffff640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f30b806be30 0x7f30b806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.020+0000 7f8561d9b640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f854406be30 0x7f854406e2f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f855c0012e0 tx=0x7f855c007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.020+0000 7f30bffff640 1 -- 172.21.15.77:0/1134961771 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f30d4106db0 con 0x7f30cc04ce40 2024-04-25T01:22:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.020+0000 7f30d37fe640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f30b806be30 0x7f30b806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.021+0000 7f30d37fe640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f30b806be30 0x7f30b806e2f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f30c8032000 tx=0x7f30c8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.021+0000 7f30bffff640 1 -- 172.21.15.77:0/1134961771 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f30d40d5410 con 0x7f30cc04ce40 2024-04-25T01:22:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.023+0000 7f37ea7fc640 1 -- 172.21.15.77:0/60631021 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f37e4021020 con 0x7f37ec04b3c0 2024-04-25T01:22:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.026+0000 7f8552ffd640 1 -- 172.21.15.77:0/1669890107 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f855807e320 con 0x7f856406d1d0 2024-04-25T01:22:10.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.028+0000 7fb58ffff640 1 -- 172.21.15.77:0/3470616456 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb598079bb0 con 0x7fb59c008cc0 2024-04-25T01:22:10.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.425+0000 7f8012977640 1 -- 172.21.15.77:0/465402458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f800c0b2ae0 con 0x7f800c1664f0 2024-04-25T01:22:10.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.425+0000 7f8008ff9640 1 -- 172.21.15.77:0/465402458 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f800005c7d0 con 0x7f800c1664f0 2024-04-25T01:22:10.406 INFO:teuthology.orchestra.run.smithi077.stdout:55834574858 2024-04-25T01:22:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.428+0000 7f7ff27fc640 1 -- 172.21.15.77:0/465402458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7ff806be30 msgr2=0x7f7ff806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.429+0000 7f7ff27fc640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7ff806be30 0x7f7ff806e2f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc005880 tx=0x7f7ffc009040 comp rx=0 tx=0).stop 2024-04-25T01:22:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.429+0000 7f7ff27fc640 1 -- 172.21.15.77:0/465402458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800c1664f0 msgr2=0x7f800c0d0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.429+0000 7f7ff27fc640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800c1664f0 0x7f800c0d0c30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f800000c3a0 tx=0x7f800000c860 comp rx=0 tx=0).stop 2024-04-25T01:22:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.429+0000 7f7ff27fc640 1 -- 172.21.15.77:0/465402458 shutdown_connections 2024-04-25T01:22:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.429+0000 7f7ff27fc640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7ff806be30 0x7f7ff806e2f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.429+0000 7f7ff27fc640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800c1664f0 0x7f800c0d0c30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.429+0000 7f7ff27fc640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f800c0c7500 0x7f800c14dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.429+0000 7f7ff27fc640 1 --2- 172.21.15.77:0/465402458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f800c06b120 0x7f800c0d0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.429+0000 7f7ff27fc640 1 -- 172.21.15.77:0/465402458 >> 172.21.15.77:0/465402458 conn(0x7f800c0b2670 msgr2=0x7f800c164a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.430+0000 7f7ff27fc640 1 -- 172.21.15.77:0/465402458 shutdown_connections 2024-04-25T01:22:10.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.430+0000 7f7ff27fc640 1 -- 172.21.15.77:0/465402458 wait complete. 2024-04-25T01:22:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.442+0000 7fc1877fe640 1 -- 172.21.15.77:0/1662845202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fc1940053c0 con 0x7fc194007460 2024-04-25T01:22:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.443+0000 7f30da11d640 1 -- 172.21.15.77:0/1134961771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f30cc005390 con 0x7f30cc04ce40 2024-04-25T01:22:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.443+0000 7fc191ffb640 1 -- 172.21.15.77:0/1662845202 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc18c05db10 con 0x7fc194007460 2024-04-25T01:22:10.424 INFO:teuthology.orchestra.run.smithi077.stdout:55834574857 2024-04-25T01:22:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.443+0000 7f30bffff640 1 -- 172.21.15.77:0/1134961771 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f30d40d4d10 con 0x7f30cc04ce40 2024-04-25T01:22:10.424 INFO:teuthology.orchestra.run.smithi077.stdout:55834574858 2024-04-25T01:22:10.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.443+0000 7f8568d29640 1 -- 172.21.15.77:0/1669890107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8548005850 con 0x7f856406d1d0 2024-04-25T01:22:10.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.444+0000 7f8552ffd640 1 -- 172.21.15.77:0/1669890107 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f855807dc20 con 0x7f856406d1d0 2024-04-25T01:22:10.425 INFO:teuthology.orchestra.run.smithi077.stdout:55834574857 2024-04-25T01:22:10.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.445+0000 7f07f2ab3640 1 -- 172.21.15.77:0/863825972 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f07e40ec090 con 0x7f07e4054390 2024-04-25T01:22:10.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.445+0000 7fdfb96ce640 1 -- 172.21.15.77:0/958613361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fdfb40734f0 con 0x7fdfb4163010 2024-04-25T01:22:10.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fdfb0ff9640 1 -- 172.21.15.77:0/958613361 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdfa4058fd0 con 0x7fdfb4163010 2024-04-25T01:22:10.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7f07e97fa640 1 -- 172.21.15.77:0/863825972 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f07d8059930 con 0x7f07e4054390 2024-04-25T01:22:10.427 INFO:teuthology.orchestra.run.smithi077.stdout:55834574859 2024-04-25T01:22:10.428 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.2 2024-04-25T01:22:10.428 INFO:teuthology.orchestra.run.smithi077.stdout:55834574859 2024-04-25T01:22:10.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fc1877fe640 1 -- 172.21.15.77:0/1662845202 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc18006bf00 msgr2=0x7fc18006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fc1877fe640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc18006bf00 0x7fc18006e3c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc1940e6d30 tx=0x7fc198009040 comp rx=0 tx=0).stop 2024-04-25T01:22:10.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fc1877fe640 1 -- 172.21.15.77:0/1662845202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc194007460 msgr2=0x7fc194185150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fc1877fe640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc194007460 0x7fc194185150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00e900 tx=0x7fc18c00edc0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb59c034860 con 0x7fb59c008cc0 2024-04-25T01:22:10.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fc1877fe640 1 -- 172.21.15.77:0/1662845202 shutdown_connections 2024-04-25T01:22:10.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fc1877fe640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc18006bf00 0x7fc18006e3c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fc1877fe640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc194054560 0x7fc19417a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fc1877fe640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc194048100 0x7fc194179bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fc1877fe640 1 --2- 172.21.15.77:0/1662845202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc194007460 0x7fc194185150 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fc1877fe640 1 -- 172.21.15.77:0/1662845202 >> 172.21.15.77:0/1662845202 conn(0x7fc194009fe0 msgr2=0x7fc1940e7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 -- 172.21.15.77:0/1134961771 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f30b806be30 msgr2=0x7f30b806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f30b806be30 0x7f30b806e2f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f30c8032000 tx=0x7f30c8008030 comp rx=0 tx=0).stop 2024-04-25T01:22:10.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 -- 172.21.15.77:0/1134961771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30cc04ce40 msgr2=0x7f30cc035690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30cc04ce40 0x7f30cc035690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f30d4064f60 tx=0x7f30d4073800 comp rx=0 tx=0).stop 2024-04-25T01:22:10.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 -- 172.21.15.77:0/1134961771 shutdown_connections 2024-04-25T01:22:10.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f30b806be30 0x7f30b806e2f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30cc035bd0 0x7f30cc0d1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30cc04ce40 0x7f30cc035690 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 --2- 172.21.15.77:0/1134961771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c4003a00 0x7f30cc04db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 -- 172.21.15.77:0/1134961771 >> 172.21.15.77:0/1134961771 conn(0x7f30cc009fb0 msgr2=0x7f30cc0db3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 -- 172.21.15.77:0/1134961771 shutdown_connections 2024-04-25T01:22:10.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.446+0000 7fc1877fe640 1 -- 172.21.15.77:0/1662845202 shutdown_connections 2024-04-25T01:22:10.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7fc1877fe640 1 -- 172.21.15.77:0/1662845202 wait complete. 2024-04-25T01:22:10.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f30da11d640 1 -- 172.21.15.77:0/1134961771 wait complete. 2024-04-25T01:22:10.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7fb58ffff640 1 -- 172.21.15.77:0/3470616456 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb59807deb0 con 0x7fb59c008cc0 2024-04-25T01:22:10.433 INFO:teuthology.orchestra.run.smithi077.stdout:55834574857 2024-04-25T01:22:10.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.447+0000 7f8568d29640 1 -- 172.21.15.77:0/1669890107 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f854406be30 msgr2=0x7f854406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.448+0000 7f8568d29640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f854406be30 0x7f854406e2f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f855c0012e0 tx=0x7f855c007d30 comp rx=0 tx=0).stop 2024-04-25T01:22:10.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.448+0000 7f8568d29640 1 -- 172.21.15.77:0/1669890107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f856406d1d0 msgr2=0x7f85642039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.448+0000 7f8568d29640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f856406d1d0 0x7f85642039a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f855802a770 tx=0x7f85580046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.448+0000 7f8568d29640 1 -- 172.21.15.77:0/1669890107 shutdown_connections 2024-04-25T01:22:10.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.448+0000 7f8568d29640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f854406be30 0x7f854406e2f0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f855c0012e0 tx=0x7f855c007d30 comp rx=0 tx=0).stop 2024-04-25T01:22:10.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.448+0000 7f8568d29640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f856406ee50 0x7f8564164210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.448+0000 7f8568d29640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85640717c0 0x7f856416ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.448+0000 7f8568d29640 1 --2- 172.21.15.77:0/1669890107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f856406d1d0 0x7f85642039a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.448+0000 7f8568d29640 1 -- 172.21.15.77:0/1669890107 >> 172.21.15.77:0/1669890107 conn(0x7f85640b1760 msgr2=0x7f85640710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.448+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f37dc005850 con 0x7f37ec04b3c0 2024-04-25T01:22:10.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.449+0000 7f8568d29640 1 -- 172.21.15.77:0/1669890107 shutdown_connections 2024-04-25T01:22:10.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.449+0000 7fdfb96ce640 1 -- 172.21.15.77:0/958613361 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdfa006bf30 msgr2=0x7fdfa006e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.449+0000 7f07f2ab3640 1 -- 172.21.15.77:0/863825972 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f07d006be30 msgr2=0x7f07d006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.449+0000 7f07f2ab3640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f07d006be30 0x7f07d006e2f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f07e0034000 tx=0x7f07e002c000 comp rx=0 tx=0).stop 2024-04-25T01:22:10.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.449+0000 7f07f2ab3640 1 -- 172.21.15.77:0/863825972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4054390 msgr2=0x7f07e40eb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.449+0000 7f07f2ab3640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4054390 0x7f07e40eb7a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f07d800d550 tx=0x7f07d800d950 comp rx=0 tx=0).stop 2024-04-25T01:22:10.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.449+0000 7fdfb96ce640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdfa006bf30 0x7fdfa006e3f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8000f60 tx=0x7fdfa8007b90 comp rx=0 tx=0).stop 2024-04-25T01:22:10.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.449+0000 7f8568d29640 1 -- 172.21.15.77:0/1669890107 wait complete. 2024-04-25T01:22:10.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.449+0000 7fdfb96ce640 1 -- 172.21.15.77:0/958613361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb4163010 msgr2=0x7fdfb4164b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.449+0000 7fdfb96ce640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb4163010 0x7fdfb4164b70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400c3f0 tx=0x7fdfa400c7f0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.449+0000 7f37ea7fc640 1 -- 172.21.15.77:0/60631021 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f37e4058f20 con 0x7f37ec04b3c0 2024-04-25T01:22:10.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fdfb96ce640 1 -- 172.21.15.77:0/958613361 shutdown_connections 2024-04-25T01:22:10.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fdfb96ce640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdfa006bf30 0x7fdfa006e3f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fdfb96ce640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb4163010 0x7fdfb4164b70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fdfb96ce640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb40c3920 0x7fdfb4162ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fdfb96ce640 1 --2- 172.21.15.77:0/958613361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfb406eec0 0x7fdfb4099a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fdfb96ce640 1 -- 172.21.15.77:0/958613361 >> 172.21.15.77:0/958613361 conn(0x7fdfb40727f0 msgr2=0x7fdfb40b26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.440 INFO:teuthology.orchestra.run.smithi077.stdout:55834574859 2024-04-25T01:22:10.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7f07f2ab3640 1 -- 172.21.15.77:0/863825972 shutdown_connections 2024-04-25T01:22:10.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7f07f2ab3640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f07d006be30 0x7f07d006e2f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7f07f2ab3640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f07e4054390 0x7f07e40eb7a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7f07f2ab3640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07e4048100 0x7f07e40eb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7f07f2ab3640 1 --2- 172.21.15.77:0/863825972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07e4007460 0x7f07e40e9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7f07f2ab3640 1 -- 172.21.15.77:0/863825972 >> 172.21.15.77:0/863825972 conn(0x7f07e4009fe0 msgr2=0x7f07e4174120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb58806be80 msgr2=0x7fb58806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fb5a9e2a640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb58806be80 0x7fb58806e340 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb59c04bf00 tx=0x7fb590007b80 comp rx=0 tx=0).stop 2024-04-25T01:22:10.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb59c008cc0 msgr2=0x7fb59c051ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fb5a9e2a640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb59c008cc0 0x7fb59c051ac0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb598035040 tx=0x7fb5980046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7f07f2ab3640 1 -- 172.21.15.77:0/863825972 shutdown_connections 2024-04-25T01:22:10.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 shutdown_connections 2024-04-25T01:22:10.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fdfb96ce640 1 -- 172.21.15.77:0/958613361 shutdown_connections 2024-04-25T01:22:10.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fb5a9e2a640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb58806be80 0x7fb58806e340 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fb5a9e2a640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb59c0e7740 0x7fb59c0e3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7f07f2ab3640 1 -- 172.21.15.77:0/863825972 wait complete. 2024-04-25T01:22:10.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fb5a9e2a640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb59c009100 0x7fb59c052000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fdfb96ce640 1 -- 172.21.15.77:0/958613361 wait complete. 2024-04-25T01:22:10.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fb5a9e2a640 1 --2- 172.21.15.77:0/3470616456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb59c008cc0 0x7fb59c051ac0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 >> 172.21.15.77:0/3470616456 conn(0x7fb59c055bd0 msgr2=0x7fb59c036690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.450+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 shutdown_connections 2024-04-25T01:22:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.451+0000 7fb5a9e2a640 1 -- 172.21.15.77:0/3470616456 wait complete. 2024-04-25T01:22:10.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.453+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f37d806beb0 msgr2=0x7f37d806e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.453+0000 7f37fa8e0640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f37d806beb0 0x7f37d806e370 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f37f4062d00 tx=0x7f37f4067eb0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.453+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ec04b3c0 msgr2=0x7f37ec045530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.453+0000 7f37fa8e0640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ec04b3c0 0x7f37ec045530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f37e400c410 tx=0x7f37e400c810 comp rx=0 tx=0).stop 2024-04-25T01:22:10.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.453+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 shutdown_connections 2024-04-25T01:22:10.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.453+0000 7f37fa8e0640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f37d806beb0 0x7f37d806e370 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f37f4062d00 tx=0x7f37f4067eb0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.453+0000 7f37fa8e0640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ec04b3c0 0x7f37ec045530 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.453+0000 7f37fa8e0640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37ec034020 0x7f37ec043ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.453+0000 7f37fa8e0640 1 --2- 172.21.15.77:0/60631021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37ec006a50 0x7f37ec0db2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.453+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 >> 172.21.15.77:0/60631021 conn(0x7f37ec00a400 msgr2=0x7f37ec0e5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.453+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 shutdown_connections 2024-04-25T01:22:10.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:10.455+0000 7f37fa8e0640 1 -- 172.21.15.77:0/60631021 wait complete. 2024-04-25T01:22:10.450 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.7 2024-04-25T01:22:10.450 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.1 2024-04-25T01:22:10.453 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.0 2024-04-25T01:22:10.454 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.4 2024-04-25T01:22:10.455 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.3 2024-04-25T01:22:10.455 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.5 2024-04-25T01:22:10.456 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.6 2024-04-25T01:22:11.429 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T01:22:11.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T01:22:11.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T01:22:11.454 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T01:22:11.458 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T01:22:11.458 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T01:22:11.459 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T01:22:11.459 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T01:22:11.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.600+0000 7f3796b70640 1 Processor -- start 2024-04-25T01:22:11.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.600+0000 7f3796b70640 1 -- start start 2024-04-25T01:22:11.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.602+0000 7f3796b70640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3788053a90 0x7f37880502a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.602+0000 7f3796b70640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3788053e70 0x7f3788054250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.602+0000 7f3796b70640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3788050d50 con 0x7f3788054980 2024-04-25T01:22:11.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.602+0000 7f3796b70640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f3788009500 con 0x7f3788053a90 2024-04-25T01:22:11.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.602+0000 7f3796b70640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3788050ff0 con 0x7f3788053e70 2024-04-25T01:22:11.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.602+0000 7f379490c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3788053a90 0x7f37880502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.603+0000 7f379490c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3788053a90 0x7f37880502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56414/0 (socket says 172.21.15.77:56414) 2024-04-25T01:22:11.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.603+0000 7f379490c640 1 -- 172.21.15.77:0/1422501068 learned_addr learned my addr 172.21.15.77:0/1422501068 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.603+0000 7f379490c640 1 -- 172.21.15.77:0/1422501068 >> v2:172.21.15.121:3300/0 conn(0x7f3788053e70 msgr2=0x7f3788054250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:22:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.603+0000 7f379490c640 1 --2- 172.21.15.77:0/1422501068 >> v2:172.21.15.121:3300/0 conn(0x7f3788053e70 0x7f3788054250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.603+0000 7f379490c640 1 -- 172.21.15.77:0/1422501068 >> v1:172.21.15.77:6789/0 conn(0x7f3788054980 legacy=0x7f3788008df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:22:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.603+0000 7f379490c640 1 -- 172.21.15.77:0/1422501068 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37880085f0 con 0x7f3788053a90 2024-04-25T01:22:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.605+0000 7f379490c640 1 --2- 172.21.15.77:0/1422501068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3788053a90 0x7f37880502a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3788056b20 tx=0x7f378402a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafa564ba688a2f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.605+0000 7f378f7fe640 1 -- 172.21.15.77:0/1422501068 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784037070 con 0x7f3788053a90 2024-04-25T01:22:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.606+0000 7f378f7fe640 1 -- 172.21.15.77:0/1422501068 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3784032440 con 0x7f3788053a90 2024-04-25T01:22:11.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.607+0000 7f3796b70640 1 -- 172.21.15.77:0/1422501068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3788053a90 msgr2=0x7f37880502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.607+0000 7f3796b70640 1 --2- 172.21.15.77:0/1422501068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3788053a90 0x7f37880502a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3788056b20 tx=0x7f378402a400 comp rx=0 tx=0).stop 2024-04-25T01:22:11.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.607+0000 7f3796b70640 1 -- 172.21.15.77:0/1422501068 shutdown_connections 2024-04-25T01:22:11.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.607+0000 7f3796b70640 1 --2- 172.21.15.77:0/1422501068 >> v2:172.21.15.121:3300/0 conn(0x7f3788053e70 0x7f3788054250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.607+0000 7f3796b70640 1 --2- 172.21.15.77:0/1422501068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3788053a90 0x7f37880502a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.607+0000 7f3796b70640 1 -- 172.21.15.77:0/1422501068 >> 172.21.15.77:0/1422501068 conn(0x7f3788055c50 msgr2=0x7f3788056060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:11.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.608+0000 7f3796b70640 1 -- 172.21.15.77:0/1422501068 shutdown_connections 2024-04-25T01:22:11.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.608+0000 7f3796b70640 1 -- 172.21.15.77:0/1422501068 wait complete. 2024-04-25T01:22:11.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.610+0000 7f3796b70640 1 Processor -- start 2024-04-25T01:22:11.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.610+0000 7f3796b70640 1 -- start start 2024-04-25T01:22:11.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.611+0000 7f3796b70640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3788053a90 0x7f37880d1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.612+0000 7f3796b70640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3788053e70 0x7f37880e2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.612+0000 7f3796b70640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3788054980 0x7f37880caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.612+0000 7f3796b70640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f37880507e0 con 0x7f3788053e70 2024-04-25T01:22:11.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.612+0000 7f3796b70640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f37880512d0 con 0x7f3788053a90 2024-04-25T01:22:11.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.612+0000 7f379490c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3788053a90 0x7f37880d1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.612+0000 7f379490c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3788053a90 0x7f37880d1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48844/0 (socket says 172.21.15.77:48844) 2024-04-25T01:22:11.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.612+0000 7f3796b70640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3788038d80 con 0x7f3788054980 2024-04-25T01:22:11.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.612+0000 7f379490c640 1 -- 172.21.15.77:0/3311640774 learned_addr learned my addr 172.21.15.77:0/3311640774 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.613+0000 7f378ffff640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3788053e70 0x7f37880e2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.613+0000 7f379490c640 1 -- 172.21.15.77:0/3311640774 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3788054980 msgr2=0x7f37880caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.613+0000 7f379490c640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3788054980 0x7f37880caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.613+0000 7f379490c640 1 -- 172.21.15.77:0/3311640774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3788053e70 msgr2=0x7f37880e2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.613+0000 7f379490c640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3788053e70 0x7f37880e2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.614+0000 7f379490c640 1 -- 172.21.15.77:0/3311640774 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37880e0f50 con 0x7f3788053a90 2024-04-25T01:22:11.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.614+0000 7f378ffff640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3788053e70 0x7f37880e2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:11.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.614+0000 7f379490c640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3788053a90 0x7f37880d1ce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3784009b90 tx=0x7f37840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.614+0000 7f378d7fa640 1 -- 172.21.15.77:0/3311640774 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784037050 con 0x7f3788053a90 2024-04-25T01:22:11.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.615+0000 7f3796b70640 1 -- 172.21.15.77:0/3311640774 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37880e1230 con 0x7f3788053a90 2024-04-25T01:22:11.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.615+0000 7f3796b70640 1 -- 172.21.15.77:0/3311640774 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37880cfaa0 con 0x7f3788053a90 2024-04-25T01:22:11.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.615+0000 7f378d7fa640 1 -- 172.21.15.77:0/3311640774 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3784038040 con 0x7f3788053a90 2024-04-25T01:22:11.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.615+0000 7f378d7fa640 1 -- 172.21.15.77:0/3311640774 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784003b20 con 0x7f3788053a90 2024-04-25T01:22:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.617+0000 7f3796b70640 1 -- 172.21.15.77:0/3311640774 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3770005290 con 0x7f3788053a90 2024-04-25T01:22:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.618+0000 7f378d7fa640 1 -- 172.21.15.77:0/3311640774 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3784045020 con 0x7f3788053a90 2024-04-25T01:22:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.618+0000 7f378d7fa640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f377406bf00 0x7f377406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.619+0000 7f378d7fa640 1 -- 172.21.15.77:0/3311640774 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f37840ab8f0 con 0x7f3788053a90 2024-04-25T01:22:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.619+0000 7f378ffff640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f377406bf00 0x7f377406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.619+0000 7f378ffff640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f377406bf00 0x7f377406e3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f37880ec070 tx=0x7f377c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.623+0000 7f378d7fa640 1 -- 172.21.15.77:0/3311640774 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3784079f50 con 0x7f3788053a90 2024-04-25T01:22:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.659+0000 7f5798829640 1 Processor -- start 2024-04-25T01:22:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.659+0000 7f5798829640 1 -- start start 2024-04-25T01:22:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.660+0000 7f5798829640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f578c057480 0x7f578c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.660+0000 7f5798829640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c04a390 0x7f578c051f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.660+0000 7f5798829640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f578c0545c0 con 0x7f578c049fb0 2024-04-25T01:22:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.660+0000 7f5798829640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f578c0513c0 con 0x7f578c057480 2024-04-25T01:22:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.660+0000 7f5798829640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f578c007c60 con 0x7f578c04a390 2024-04-25T01:22:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.661+0000 7f579359e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c04a390 0x7f578c051f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.661+0000 7f98fffff640 1 Processor -- start 2024-04-25T01:22:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.661+0000 7f579359e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c04a390 0x7f578c051f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33888/0 (socket says 172.21.15.77:33888) 2024-04-25T01:22:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.661+0000 7f579359e640 1 -- 172.21.15.77:0/1517300457 learned_addr learned my addr 172.21.15.77:0/1517300457 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.661+0000 7f579259c640 1 --2- 172.21.15.77:0/1517300457 >> v2:172.21.15.77:3300/0 conn(0x7f578c057480 0x7f578c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.661+0000 7f579359e640 1 -- 172.21.15.77:0/1517300457 >> v2:172.21.15.77:3300/0 conn(0x7f578c057480 msgr2=0x7f578c036db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.661+0000 7f579359e640 1 --2- 172.21.15.77:0/1517300457 >> v2:172.21.15.77:3300/0 conn(0x7f578c057480 0x7f578c036db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.661+0000 7f579359e640 1 -- 172.21.15.77:0/1517300457 >> v1:172.21.15.77:6789/0 conn(0x7f578c049fb0 legacy=0x7f578c051850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:22:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.661+0000 7f579359e640 1 -- 172.21.15.77:0/1517300457 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578c007f40 con 0x7f578c04a390 2024-04-25T01:22:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.661+0000 7f98fffff640 1 -- start start 2024-04-25T01:22:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.662+0000 7f3b24eff640 1 Processor -- start 2024-04-25T01:22:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.662+0000 7f579259c640 1 --2- 172.21.15.77:0/1517300457 >> v2:172.21.15.77:3300/0 conn(0x7f578c057480 0x7f578c036db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.662+0000 7fbeaf5b0640 1 Processor -- start 2024-04-25T01:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.662+0000 7f98fffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98f80df220 0x7f98f803a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.662+0000 7fbeaf5b0640 1 -- start start 2024-04-25T01:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.662+0000 7f98fffff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f98f80e2d80 0x7f98f80e31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98fffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98f8004b00 0x7f98f8053c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98fffff640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f98f80e3720 con 0x7f98f80df220 2024-04-25T01:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98fffff640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f98f8054640 con 0x7f98f80e2d80 2024-04-25T01:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98fffff640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f98f8007bf0 con 0x7f98f8004b00 2024-04-25T01:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98feffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98f80df220 0x7f98f803a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98feffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98f80df220 0x7f98f803a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56438/0 (socket says 172.21.15.77:56438) 2024-04-25T01:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98feffd640 1 -- 172.21.15.77:0/1563157026 learned_addr learned my addr 172.21.15.77:0/1563157026 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98ff7fe640 1 --2- 172.21.15.77:0/1563157026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98f8004b00 0x7f98f8053c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98feffd640 1 -- 172.21.15.77:0/1563157026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98f8004b00 msgr2=0x7f98f8053c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98feffd640 1 --2- 172.21.15.77:0/1563157026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98f8004b00 0x7f98f8053c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98feffd640 1 -- 172.21.15.77:0/1563157026 >> v2:172.21.15.121:3300/0 conn(0x7f98f80e2d80 msgr2=0x7f98f80e31e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98feffd640 1 --2- 172.21.15.77:0/1563157026 >> v2:172.21.15.121:3300/0 conn(0x7f98f80e2d80 0x7f98f80e31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f98feffd640 1 -- 172.21.15.77:0/1563157026 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f8054920 con 0x7f98f80df220 2024-04-25T01:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f579359e640 1 --2- 172.21.15.77:0/1517300457 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c04a390 0x7f578c051f60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f579407ad50 tx=0x7f57940a5d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38fae4e1dba68d13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.662+0000 7f3b24eff640 1 -- start start 2024-04-25T01:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f3b24eff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b20157930 0x7f3b2006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f3b24eff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b2006d0b0 0x7f3b200ae950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f3b24eff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b2015a7e0 con 0x7f3b200aee90 2024-04-25T01:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f3b24eff640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f3b20070b10 con 0x7f3b2006d0b0 2024-04-25T01:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f3b24eff640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3b200af2c0 con 0x7f3b20157930 2024-04-25T01:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f5791d9b640 1 -- 172.21.15.77:0/1517300457 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57940b3070 con 0x7f578c04a390 2024-04-25T01:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f862759e640 1 Processor -- start 2024-04-25T01:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7f5791d9b640 1 -- 172.21.15.77:0/1517300457 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57940a6ae0 con 0x7f578c04a390 2024-04-25T01:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7f3b1f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b20157930 0x7f3b2006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7f3b1f7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b20157930 0x7f3b2006cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33906/0 (socket says 172.21.15.77:33906) 2024-04-25T01:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7f3b1f7fe640 1 -- 172.21.15.77:0/3642413933 learned_addr learned my addr 172.21.15.77:0/3642413933 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7fbeaf5b0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbea80b7ad0 0x7fbea80d2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7fbeaf5b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbea80b34f0 0x7fbea80d2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7fbeaf5b0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbea80b2f80 0x7fbea809bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7f5bc54e0640 1 Processor -- start 2024-04-25T01:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7f5bc54e0640 1 -- start start 2024-04-25T01:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7fbeaf5b0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fbea80d2a80 con 0x7fbea80b2f80 2024-04-25T01:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7f5791d9b640 1 -- 172.21.15.77:0/1517300457 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57940ae440 con 0x7f578c04a390 2024-04-25T01:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.663+0000 7f862759e640 1 -- start start 2024-04-25T01:22:11.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7f98feffd640 1 --2- 172.21.15.77:0/1563157026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f80df220 0x7f98f803a7f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f990007ad50 tx=0x7f99000a5d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0cede8037ce9b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7fbeadb4d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbea80b2f80 0x7fbea809bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7fbeadb4d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbea80b2f80 0x7fbea809bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56464/0 (socket says 172.21.15.77:56464) 2024-04-25T01:22:11.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7fbeadb4d640 1 -- 172.21.15.77:0/3920289021 learned_addr learned my addr 172.21.15.77:0/3920289021 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3920289021 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fbea806d6e0 con 0x7fbea80b7ad0 2024-04-25T01:22:11.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7f3b1f7fe640 1 -- 172.21.15.77:0/3642413933 >> v2:172.21.15.77:3300/0 conn(0x7f3b2006d0b0 msgr2=0x7f3b200ae950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:22:11.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3920289021 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbea80b6dc0 con 0x7fbea80b34f0 2024-04-25T01:22:11.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7fbead34c640 1 --2- 172.21.15.77:0/3920289021 >> v2:172.21.15.121:3300/0 conn(0x7fbea80b7ad0 0x7fbea80d2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7fbeacb4b640 1 --2- 172.21.15.77:0/3920289021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbea80b34f0 0x7fbea80d2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.664+0000 7f3b1effd640 1 --2- 172.21.15.77:0/3642413933 >> v2:172.21.15.77:3300/0 conn(0x7f3b2006d0b0 0x7f3b200ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f5798829640 1 -- 172.21.15.77:0/1517300457 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c04a390 msgr2=0x7f578c051f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f5798829640 1 --2- 172.21.15.77:0/1517300457 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c04a390 0x7f578c051f60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f579407ad50 tx=0x7f57940a5d10 comp rx=0 tx=0).stop 2024-04-25T01:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f5798829640 1 -- 172.21.15.77:0/1517300457 shutdown_connections 2024-04-25T01:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f5798829640 1 --2- 172.21.15.77:0/1517300457 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c04a390 0x7f578c051f60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f5798829640 1 --2- 172.21.15.77:0/1517300457 >> v2:172.21.15.77:3300/0 conn(0x7f578c057480 0x7f578c036db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f5798829640 1 -- 172.21.15.77:0/1517300457 >> 172.21.15.77:0/1517300457 conn(0x7f578c020040 msgr2=0x7f578c056d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:11.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f98fdffb640 1 -- 172.21.15.77:0/1563157026 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99000b3070 con 0x7f98f80df220 2024-04-25T01:22:11.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f98fdffb640 1 -- 172.21.15.77:0/1563157026 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99000a6b50 con 0x7f98f80df220 2024-04-25T01:22:11.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f98fdffb640 1 -- 172.21.15.77:0/1563157026 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99000ae440 con 0x7f98f80df220 2024-04-25T01:22:11.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f862759e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86280d09e0 0x7f86280b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f3b1f7fe640 1 --2- 172.21.15.77:0/3642413933 >> v2:172.21.15.77:3300/0 conn(0x7f3b2006d0b0 0x7f3b200ae950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f3b1f7fe640 1 -- 172.21.15.77:0/3642413933 >> v1:172.21.15.121:6789/0 conn(0x7f3b200aee90 legacy=0x7f3b2015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f3b1f7fe640 1 -- 172.21.15.77:0/3642413933 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2006d6e0 con 0x7f3b20157930 2024-04-25T01:22:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f5bc54e0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0161550 0x7f5bc00d28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f5bc54e0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5bc00ce910 0x7f5bc00b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f5bc54e0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5bc00b3b10 0x7f5bc00b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f862659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86280d09e0 0x7f86280b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7f862659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86280d09e0 0x7f86280b2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56478/0 (socket says 172.21.15.77:56478) 2024-04-25T01:22:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f5bc54e0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc00b5770 con 0x7f5bc00ce910 2024-04-25T01:22:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f5bc54e0640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc00b4670 con 0x7f5bc00b3b10 2024-04-25T01:22:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f5bc54e0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc00b4950 con 0x7f5bc0161550 2024-04-25T01:22:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f98fffff640 1 -- 172.21.15.77:0/1563157026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f80df220 msgr2=0x7f98f803a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f98fffff640 1 --2- 172.21.15.77:0/1563157026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f80df220 0x7f98f803a7f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f990007ad50 tx=0x7f99000a5d80 comp rx=0 tx=0).stop 2024-04-25T01:22:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f98fffff640 1 -- 172.21.15.77:0/1563157026 shutdown_connections 2024-04-25T01:22:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f98fffff640 1 --2- 172.21.15.77:0/1563157026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98f8004b00 0x7f98f8053c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f98fffff640 1 --2- 172.21.15.77:0/1563157026 >> v2:172.21.15.121:3300/0 conn(0x7f98f80e2d80 0x7f98f80e31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f98fffff640 1 --2- 172.21.15.77:0/1563157026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f80df220 0x7f98f803a7f0 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f990007ad50 tx=0x7f99000a5d80 comp rx=0 tx=0).stop 2024-04-25T01:22:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f98fffff640 1 -- 172.21.15.77:0/1563157026 >> 172.21.15.77:0/1563157026 conn(0x7f98f800a400 msgr2=0x7f98f800a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f3b1effd640 1 --2- 172.21.15.77:0/3642413933 >> v2:172.21.15.77:3300/0 conn(0x7f3b2006d0b0 0x7f3b200ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f3b1f7fe640 1 --2- 172.21.15.77:0/3642413933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b20157930 0x7f3b2006cb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800d370 tx=0x7f3b1802cb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=354e4f04fb13206c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f5bbffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0161550 0x7f5bc00d28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f5bc4cdf640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5bc00b3b10 0x7f5bc00b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7fbeadb4d640 1 -- 172.21.15.77:0/3920289021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbea80b34f0 msgr2=0x7fbea80d2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7fbeadb4d640 1 --2- 172.21.15.77:0/3920289021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbea80b34f0 0x7fbea80d2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7fbeadb4d640 1 -- 172.21.15.77:0/3920289021 >> v2:172.21.15.121:3300/0 conn(0x7fbea80b7ad0 msgr2=0x7fbea80d2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7fbeadb4d640 1 --2- 172.21.15.77:0/3920289021 >> v2:172.21.15.121:3300/0 conn(0x7fbea80b7ad0 0x7fbea80d2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.665+0000 7fbeadb4d640 1 -- 172.21.15.77:0/3920289021 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea80b70a0 con 0x7fbea80b2f80 2024-04-25T01:22:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7fb558837640 1 Processor -- start 2024-04-25T01:22:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f5798829640 1 -- 172.21.15.77:0/1517300457 shutdown_connections 2024-04-25T01:22:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7fb558837640 1 -- start start 2024-04-25T01:22:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f5bbffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0161550 0x7f5bc00d28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33926/0 (socket says 172.21.15.77:33926) 2024-04-25T01:22:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f5bbffff640 1 -- 172.21.15.77:0/3024222212 learned_addr learned my addr 172.21.15.77:0/3024222212 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f862759e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86280c7580 con 0x7f86280cffb0 2024-04-25T01:22:11.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f862759e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86280d46f0 con 0x7f86280d2750 2024-04-25T01:22:11.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f8625d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f86280cffb0 0x7f86280b41b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56538/0 (socket says 172.21.15.77:56538) 2024-04-25T01:22:11.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f8625d9b640 1 -- 172.21.15.77:0/1017057998 learned_addr learned my addr 172.21.15.77:0/1017057998 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:11.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f862759e640 1 -- 172.21.15.77:0/1017057998 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f86280c7330 con 0x7f86280d09e0 2024-04-25T01:22:11.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 51621032 0 0) 0x7f86280d46f0 con 0x7f86280d2750 2024-04-25T01:22:11.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8610003520 con 0x7f86280d2750 2024-04-25T01:22:11.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f5bbf7fe640 1 --2- 172.21.15.77:0/3024222212 >> v2:172.21.15.77:3300/0 conn(0x7f5bc00ce910 0x7f5bc00b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7fbeadb4d640 1 --2- 172.21.15.77:0/3920289021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea80b2f80 0x7fbea809bdf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbea80d1a70 tx=0x7fbea002a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d07b37a5511f057 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3558303305 0 0) 0x7f8610003520 con 0x7f86280d2750 2024-04-25T01:22:11.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f98fffff640 1 -- 172.21.15.77:0/1563157026 shutdown_connections 2024-04-25T01:22:11.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f5798829640 1 -- 172.21.15.77:0/1517300457 wait complete. 2024-04-25T01:22:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f3b1e7fc640 1 -- 172.21.15.77:0/3642413933 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18036070 con 0x7f3b20157930 2024-04-25T01:22:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 >> v2:172.21.15.77:3300/0 conn(0x7f86280d09e0 msgr2=0x7f86280b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f862559a640 1 --2- 172.21.15.77:0/1017057998 >> v2:172.21.15.77:3300/0 conn(0x7f86280d09e0 0x7f86280b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 >> v1:172.21.15.77:6789/0 conn(0x7f86280cffb0 legacy=0x7f86280b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86280d49d0 con 0x7f86280d2750 2024-04-25T01:22:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.666+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f8620001920 con 0x7f86280d2750 2024-04-25T01:22:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 >> v1:172.21.15.121:6789/0 conn(0x7f86280d2750 legacy=0x7f86280d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862559a640 1 --2- 172.21.15.77:0/1017057998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8610003a00 0x7f8610003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862559a640 1 --2- 172.21.15.77:0/1017057998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8610004800 0x7f8610004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862559a640 1 --2- 172.21.15.77:0/1017057998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86100051c0 0x7f861000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86280c7330 con 0x7f8610003a00 2024-04-25T01:22:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f861000c0c0 con 0x7f8610004800 2024-04-25T01:22:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f861000c2c0 con 0x7f86100051c0 2024-04-25T01:22:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f8626d9d640 1 --2- 172.21.15.77:0/1017057998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86100051c0 0x7f861000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862659c640 1 --2- 172.21.15.77:0/1017057998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8610003a00 0x7f8610003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f3b1e7fc640 1 -- 172.21.15.77:0/3642413933 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b1802e860 con 0x7f3b20157930 2024-04-25T01:22:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f5bbffff640 1 -- 172.21.15.77:0/3024222212 >> v2:172.21.15.121:3300/0 conn(0x7f5bc00b3b10 msgr2=0x7f5bc00b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f5bbffff640 1 --2- 172.21.15.77:0/3024222212 >> v2:172.21.15.121:3300/0 conn(0x7f5bc00b3b10 0x7f5bc00b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f5bbffff640 1 -- 172.21.15.77:0/3024222212 >> v2:172.21.15.77:3300/0 conn(0x7f5bc00ce910 msgr2=0x7f5bc00b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f5bbffff640 1 --2- 172.21.15.77:0/3024222212 >> v2:172.21.15.77:3300/0 conn(0x7f5bc00ce910 0x7f5bc00b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f5bbffff640 1 -- 172.21.15.77:0/3024222212 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc00b4c30 con 0x7f5bc0161550 2024-04-25T01:22:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7fbe9ffff640 1 -- 172.21.15.77:0/3920289021 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0036070 con 0x7fbea80b2f80 2024-04-25T01:22:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7fbe9ffff640 1 -- 172.21.15.77:0/3920289021 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea0031600 con 0x7fbea80b2f80 2024-04-25T01:22:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7fbe9ffff640 1 -- 172.21.15.77:0/3920289021 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea00319d0 con 0x7fbea80b2f80 2024-04-25T01:22:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7fb558837640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb5540cf520 0x7fb5540ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7fb558837640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55406d910 0x7fb5540aec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7fb558837640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb55406ba90 con 0x7fb55415ad40 2024-04-25T01:22:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7fb558837640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb55406b830 con 0x7fb5540cf520 2024-04-25T01:22:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7fb558837640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb55406cb50 con 0x7fb55406d910 2024-04-25T01:22:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f98fffff640 1 -- 172.21.15.77:0/1563157026 wait complete. 2024-04-25T01:22:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7f3b24eff640 1 -- 172.21.15.77:0/3642413933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b20157930 msgr2=0x7f3b2006cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fb55359e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55406d910 0x7fb5540aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fb55359e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55406d910 0x7fb5540aec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33944/0 (socket says 172.21.15.77:33944) 2024-04-25T01:22:11.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fb55359e640 1 -- 172.21.15.77:0/1188251837 learned_addr learned my addr 172.21.15.77:0/1188251837 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7f3b24eff640 1 --2- 172.21.15.77:0/3642413933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b20157930 0x7f3b2006cb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800d370 tx=0x7f3b1802cb00 comp rx=0 tx=0).stop 2024-04-25T01:22:11.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862659c640 1 -- 172.21.15.77:0/1017057998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86100051c0 msgr2=0x7f861000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:11.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862659c640 1 --2- 172.21.15.77:0/1017057998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86100051c0 0x7f861000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862659c640 1 -- 172.21.15.77:0/1017057998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8610004800 msgr2=0x7f8610004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:11.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862659c640 1 --2- 172.21.15.77:0/1017057998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8610004800 0x7f8610004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.667+0000 7f862659c640 1 -- 172.21.15.77:0/1017057998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86280d49d0 con 0x7f8610003a00 2024-04-25T01:22:11.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7f8626d9d640 1 -- 172.21.15.77:0/1017057998 reap_dead start 2024-04-25T01:22:11.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7f862659c640 1 --2- 172.21.15.77:0/1017057998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8610003a00 0x7f8610003e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8618009aa0 tx=0x7f8610005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b1c8020cbe926b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7f98fffff640 1 Processor -- start 2024-04-25T01:22:11.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7f98fffff640 1 -- start start 2024-04-25T01:22:11.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7f5bbffff640 1 --2- 172.21.15.77:0/3024222212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0161550 0x7f5bc00d28f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5bc008c090 tx=0x7f5bb802a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8956f23b3150b78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7f5bbeffd640 1 -- 172.21.15.77:0/3024222212 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8036070 con 0x7f5bc0161550 2024-04-25T01:22:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7f5bbeffd640 1 -- 172.21.15.77:0/3024222212 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb8031600 con 0x7f5bc0161550 2024-04-25T01:22:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7f5bbeffd640 1 -- 172.21.15.77:0/3024222212 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8031a10 con 0x7f5bc0161550 2024-04-25T01:22:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3920289021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea80b2f80 msgr2=0x7fbea809bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fbeaf5b0640 1 --2- 172.21.15.77:0/3920289021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea80b2f80 0x7fbea809bdf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbea80d1a70 tx=0x7fbea002a690 comp rx=0 tx=0).stop 2024-04-25T01:22:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3920289021 shutdown_connections 2024-04-25T01:22:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fbeaf5b0640 1 --2- 172.21.15.77:0/3920289021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea80b2f80 0x7fbea809bdf0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fbeaf5b0640 1 --2- 172.21.15.77:0/3920289021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbea80b34f0 0x7fbea80d2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fbeaf5b0640 1 --2- 172.21.15.77:0/3920289021 >> v2:172.21.15.121:3300/0 conn(0x7fbea80b7ad0 0x7fbea80d2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3920289021 >> 172.21.15.77:0/3920289021 conn(0x7fbea80d09f0 msgr2=0x7fbea80d0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3920289021 shutdown_connections 2024-04-25T01:22:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3920289021 wait complete. 2024-04-25T01:22:11.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7f5798829640 1 Processor -- start 2024-04-25T01:22:11.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7f5798829640 1 -- start start 2024-04-25T01:22:11.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fb55359e640 1 -- 172.21.15.77:0/1188251837 >> v2:172.21.15.121:3300/0 conn(0x7fb5540cf520 msgr2=0x7fb5540ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:11.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fb55359e640 1 --2- 172.21.15.77:0/1188251837 >> v2:172.21.15.121:3300/0 conn(0x7fb5540cf520 0x7fb5540ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fb55359e640 1 -- 172.21.15.77:0/1188251837 >> v1:172.21.15.121:6789/0 conn(0x7fb55415ad40 legacy=0x7fb55406b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:11.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7fb55359e640 1 -- 172.21.15.77:0/1188251837 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb55406ce30 con 0x7fb55406d910 2024-04-25T01:22:11.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7f3b24eff640 1 -- 172.21.15.77:0/3642413933 shutdown_connections 2024-04-25T01:22:11.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.668+0000 7f3b24eff640 1 --2- 172.21.15.77:0/3642413933 >> v2:172.21.15.77:3300/0 conn(0x7f3b2006d0b0 0x7f3b200ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618036070 con 0x7f8610003a00 2024-04-25T01:22:11.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8618031400 con 0x7f8610003a00 2024-04-25T01:22:11.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f862559a640 1 -- 172.21.15.77:0/1017057998 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618009670 con 0x7f8610003a00 2024-04-25T01:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5798829640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c049fb0 0x7f578c0da2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5798829640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f578c04a390 0x7f578c050aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7fb55359e640 1 --2- 172.21.15.77:0/1188251837 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55406d910 0x7fb5540aec70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb54400d150 tx=0x7fb54402c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3163463413a126f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7fb551d9b640 1 -- 172.21.15.77:0/1188251837 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb544009600 con 0x7fb55406d910 2024-04-25T01:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7fb551d9b640 1 -- 172.21.15.77:0/1188251837 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5440097a0 con 0x7fb55406d910 2024-04-25T01:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7f3b24eff640 1 --2- 172.21.15.77:0/3642413933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b20157930 0x7f3b2006cb70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7f3b24eff640 1 -- 172.21.15.77:0/3642413933 >> 172.21.15.77:0/3642413933 conn(0x7f3b2006b7e0 msgr2=0x7f3b200b1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:11.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7f3b24eff640 1 -- 172.21.15.77:0/3642413933 shutdown_connections 2024-04-25T01:22:11.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.669+0000 7f3b24eff640 1 -- 172.21.15.77:0/3642413933 wait complete. 2024-04-25T01:22:11.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5798829640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f578c057480 0x7f578c035e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f579259c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f578c04a390 0x7f578c050aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5bc54e0640 1 -- 172.21.15.77:0/3024222212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0161550 msgr2=0x7f5bc00d28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5bc54e0640 1 --2- 172.21.15.77:0/3024222212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0161550 0x7f5bc00d28f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5bc008c090 tx=0x7f5bb802a6d0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f579259c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f578c04a390 0x7f578c050aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48906/0 (socket says 172.21.15.77:48906) 2024-04-25T01:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5bc54e0640 1 -- 172.21.15.77:0/3024222212 shutdown_connections 2024-04-25T01:22:11.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7fb551d9b640 1 -- 172.21.15.77:0/1188251837 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb544009bb0 con 0x7fb55406d910 2024-04-25T01:22:11.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5bc54e0640 1 --2- 172.21.15.77:0/3024222212 >> v2:172.21.15.121:3300/0 conn(0x7f5bc00b3b10 0x7f5bc00b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f579259c640 1 -- 172.21.15.77:0/1577492919 learned_addr learned my addr 172.21.15.77:0/1577492919 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5bc54e0640 1 --2- 172.21.15.77:0/3024222212 >> v2:172.21.15.77:3300/0 conn(0x7f5bc00ce910 0x7f5bc00b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5bc54e0640 1 --2- 172.21.15.77:0/3024222212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0161550 0x7f5bc00d28f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f578c0372f0 con 0x7f578c057480 2024-04-25T01:22:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5bc54e0640 1 -- 172.21.15.77:0/3024222212 >> 172.21.15.77:0/3024222212 conn(0x7f5bc009d2b0 msgr2=0x7f5bc00d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f578c038580 con 0x7f578c04a390 2024-04-25T01:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f98fffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f8004b00 0x7f98f804ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f578c037470 con 0x7f578c049fb0 2024-04-25T01:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f579359e640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f578c057480 0x7f578c035e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f98fffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98f80e2d80 0x7f98f804d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7fb558837640 1 -- 172.21.15.77:0/1188251837 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55406d910 msgr2=0x7fb5540aec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7fb558837640 1 --2- 172.21.15.77:0/1188251837 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55406d910 0x7fb5540aec70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb54400d150 tx=0x7fb54402c900 comp rx=0 tx=0).stop 2024-04-25T01:22:11.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f98fffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98f804c9f0 0x7f98f804a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f98fffff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f98f80065c0 con 0x7f98f8004b00 2024-04-25T01:22:11.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f98fffff640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f98f80062e0 con 0x7f98f804c9f0 2024-04-25T01:22:11.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f98fffff640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f98f80df600 con 0x7f98f80e2d80 2024-04-25T01:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7fb558837640 1 -- 172.21.15.77:0/1188251837 shutdown_connections 2024-04-25T01:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7fb558837640 1 --2- 172.21.15.77:0/1188251837 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55406d910 0x7fb5540aec70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7fb558837640 1 --2- 172.21.15.77:0/1188251837 >> v2:172.21.15.121:3300/0 conn(0x7fb5540cf520 0x7fb5540ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7fb558837640 1 -- 172.21.15.77:0/1188251837 >> 172.21.15.77:0/1188251837 conn(0x7fb5540b1660 msgr2=0x7fb5540b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f98feffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f8004b00 0x7f98f804ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.670+0000 7f5792d9d640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c049fb0 0x7f578c0da2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7fbeaf5b0640 1 Processor -- start 2024-04-25T01:22:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7fb558837640 1 -- 172.21.15.77:0/1188251837 shutdown_connections 2024-04-25T01:22:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98f804c9f0 0x7f98f804a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7fbeaf5b0640 1 -- start start 2024-04-25T01:22:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f862759e640 1 -- 172.21.15.77:0/1017057998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8610003a00 msgr2=0x7f8610003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f862759e640 1 --2- 172.21.15.77:0/1017057998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8610003a00 0x7f8610003e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8618009aa0 tx=0x7f8610005830 comp rx=0 tx=0).stop 2024-04-25T01:22:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f98feffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f8004b00 0x7f98f804ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56510/0 (socket says 172.21.15.77:56510) 2024-04-25T01:22:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f98feffd640 1 -- 172.21.15.77:0/3685311099 learned_addr learned my addr 172.21.15.77:0/3685311099 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f579259c640 1 -- 172.21.15.77:0/1577492919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c049fb0 msgr2=0x7f578c0da2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f579259c640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c049fb0 0x7f578c0da2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f579259c640 1 -- 172.21.15.77:0/1577492919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f578c057480 msgr2=0x7f578c035e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f579259c640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f578c057480 0x7f578c035e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f579259c640 1 -- 172.21.15.77:0/1577492919 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578c0da890 con 0x7f578c04a390 2024-04-25T01:22:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f862759e640 1 -- 172.21.15.77:0/1017057998 shutdown_connections 2024-04-25T01:22:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f862759e640 1 --2- 172.21.15.77:0/1017057998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8610003a00 0x7f8610003e20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f862759e640 1 -- 172.21.15.77:0/1017057998 >> 172.21.15.77:0/1017057998 conn(0x7f862809d170 msgr2=0x7f86280d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f579259c640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f578c04a390 0x7f578c050aa0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f578400ebe0 tx=0x7f578400c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f862759e640 1 -- 172.21.15.77:0/1017057998 shutdown_connections 2024-04-25T01:22:11.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f98feffd640 1 -- 172.21.15.77:0/3685311099 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98f80e2d80 msgr2=0x7f98f804d250 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:22:11.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f5bc54e0640 1 -- 172.21.15.77:0/3024222212 shutdown_connections 2024-04-25T01:22:11.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f98feffd640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98f80e2d80 0x7f98f804d250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f98feffd640 1 -- 172.21.15.77:0/3685311099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98f804c9f0 msgr2=0x7f98f804a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f577b7fe640 1 -- 172.21.15.77:0/1577492919 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784017070 con 0x7f578c04a390 2024-04-25T01:22:11.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f98feffd640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98f804c9f0 0x7f98f804a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7f577b7fe640 1 -- 172.21.15.77:0/1577492919 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f578400cc30 con 0x7f578c04a390 2024-04-25T01:22:11.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f98feffd640 1 -- 172.21.15.77:0/3685311099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f804ce70 con 0x7f98f8004b00 2024-04-25T01:22:11.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7f577b7fe640 1 -- 172.21.15.77:0/1577492919 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784017070 con 0x7f578c04a390 2024-04-25T01:22:11.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7f98feffd640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f8004b00 0x7f98f804ea20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f990006ac00 tx=0x7f990006b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f578c034390 con 0x7f578c04a390 2024-04-25T01:22:11.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7f862759e640 1 -- 172.21.15.77:0/1017057998 wait complete. 2024-04-25T01:22:11.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7f862759e640 1 Processor -- start 2024-04-25T01:22:11.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7f98e77fe640 1 -- 172.21.15.77:0/3685311099 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99000b3040 con 0x7f98f8004b00 2024-04-25T01:22:11.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7f98e77fe640 1 -- 172.21.15.77:0/3685311099 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99000be070 con 0x7f98f8004b00 2024-04-25T01:22:11.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7f98fffff640 1 -- 172.21.15.77:0/3685311099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98f804d980 con 0x7f98f8004b00 2024-04-25T01:22:11.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7f3b24eff640 1 Processor -- start 2024-04-25T01:22:11.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7f3b24eff640 1 -- start start 2024-04-25T01:22:11.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7f3b24eff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b2006d0b0 0x7f3b201566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7f3b24eff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b200aee90 0x7f3b2014cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7f3b24eff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b20157930 0x7f3b2014d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7f3b24eff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3b20070d60 con 0x7f3b200aee90 2024-04-25T01:22:11.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7f5bc54e0640 1 -- 172.21.15.77:0/3024222212 wait complete. 2024-04-25T01:22:11.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7fbeaf5b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea80b2f80 0x7fbea81ead00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7fbead34c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea80b2f80 0x7fbea81ead00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7fbead34c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea80b2f80 0x7fbea81ead00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56518/0 (socket says 172.21.15.77:56518) 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7fbead34c640 1 -- 172.21.15.77:0/3170279757 learned_addr learned my addr 172.21.15.77:0/3170279757 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7fbeaf5b0640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea80b34f0 0x7fbea80b6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.672+0000 7fbeaf5b0640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbea80b7ad0 0x7fbea81ea550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3170279757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbea80b7eb0 con 0x7fbea80b2f80 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3170279757 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbea80d2cd0 con 0x7fbea80b34f0 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7fbeadb4d640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbea80b7ad0 0x7fbea81ea550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3170279757 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbea80b8030 con 0x7fbea80b7ad0 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7fbead34c640 1 -- 172.21.15.77:0/3170279757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbea80b7ad0 msgr2=0x7fbea81ea550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7fbead34c640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbea80b7ad0 0x7fbea81ea550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7fbead34c640 1 -- 172.21.15.77:0/3170279757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea80b34f0 msgr2=0x7fbea80b6360 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7fbead34c640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea80b34f0 0x7fbea80b6360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7fbead34c640 1 -- 172.21.15.77:0/3170279757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea80ca6b0 con 0x7fbea80b2f80 2024-04-25T01:22:11.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.671+0000 7fb558837640 1 -- 172.21.15.77:0/1188251837 wait complete. 2024-04-25T01:22:11.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7fb558837640 1 Processor -- start 2024-04-25T01:22:11.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7f3b1ffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b20157930 0x7f3b2014d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7f3b1ffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b20157930 0x7f3b2014d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48930/0 (socket says 172.21.15.77:48930) 2024-04-25T01:22:11.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7f3b1ffff640 1 -- 172.21.15.77:0/2936255665 learned_addr learned my addr 172.21.15.77:0/2936255665 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2006dca0 con 0x7f3b20157930 2024-04-25T01:22:11.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3b20157d10 con 0x7f3b2006d0b0 2024-04-25T01:22:11.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7f3b1f7fe640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b2006d0b0 0x7f3b201566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7fbead34c640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea80b2f80 0x7fbea81ead00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbe9800c850 tx=0x7fbe9800cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7f3b1f7fe640 1 -- 172.21.15.77:0/2936255665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b20157930 msgr2=0x7f3b2014d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7f5bc54e0640 1 Processor -- start 2024-04-25T01:22:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7fbe9dffb640 1 -- 172.21.15.77:0/3170279757 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe98015070 con 0x7fbea80b2f80 2024-04-25T01:22:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7fb558837640 1 -- start start 2024-04-25T01:22:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7fb558837640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb55406d910 0x7fb55415b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7f3b1f7fe640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b20157930 0x7f3b2014d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7fbe9dffb640 1 -- 172.21.15.77:0/3170279757 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe98004220 con 0x7fbea80b2f80 2024-04-25T01:22:11.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7fbe9dffb640 1 -- 172.21.15.77:0/3170279757 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe98007620 con 0x7fbea80b2f80 2024-04-25T01:22:11.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f3b1f7fe640 1 -- 172.21.15.77:0/2936255665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b200aee90 msgr2=0x7f3b2014cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3170279757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea80b5ff0 con 0x7fbea80b2f80 2024-04-25T01:22:11.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f5bc54e0640 1 -- start start 2024-04-25T01:22:11.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f3b1effd640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b200aee90 0x7f3b2014cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f3b1f7fe640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b200aee90 0x7f3b2014cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f3b1f7fe640 1 -- 172.21.15.77:0/2936255665 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b201535d0 con 0x7f3b2006d0b0 2024-04-25T01:22:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7fbeaf5b0640 1 -- 172.21.15.77:0/3170279757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea81eaa90 con 0x7fbea80b2f80 2024-04-25T01:22:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f3b1ffff640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b20157930 0x7f3b2014d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.674+0000 7f862759e640 1 -- start start 2024-04-25T01:22:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f862759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8610003a00 0x7f86280ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f862759e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280ca1e0 0x7f86280c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f862759e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f862815d0e0 0x7f862815d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f862759e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86280b3140 con 0x7f8610003a00 2024-04-25T01:22:11.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f862759e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f86280b2fc0 con 0x7f86280ca1e0 2024-04-25T01:22:11.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f862759e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f86280d2b60 con 0x7f862815d0e0 2024-04-25T01:22:11.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7f98fffff640 1 -- 172.21.15.77:0/3685311099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98f804df40 con 0x7f98f8004b00 2024-04-25T01:22:11.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f3b1effd640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b200aee90 0x7f3b2014cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:11.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.675+0000 7f3b1f7fe640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b2006d0b0 0x7f3b201566b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3b18035000 tx=0x7f3b18004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.673+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f578c034950 con 0x7f578c04a390 2024-04-25T01:22:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f5bc54e0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bc00b3b10 0x7f5bc00cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f3b07fff640 1 -- 172.21.15.77:0/2936255665 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18036040 con 0x7f3b2006d0b0 2024-04-25T01:22:11.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f577b7fe640 1 -- 172.21.15.77:0/1577492919 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f578401a070 con 0x7f578c04a390 2024-04-25T01:22:11.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f578c0057e0 con 0x7f578c04a390 2024-04-25T01:22:11.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f3b07fff640 1 -- 172.21.15.77:0/2936255665 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b18041070 con 0x7f3b2006d0b0 2024-04-25T01:22:11.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b201538b0 con 0x7f3b2006d0b0 2024-04-25T01:22:11.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f5bc54e0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5bc00ce910 0x7f5bc015e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f5bc54e0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0161550 0x7f5bc00b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f5bc54e0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc0161930 con 0x7f5bc00b3b10 2024-04-25T01:22:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f5bc54e0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc00b5280 con 0x7f5bc00ce910 2024-04-25T01:22:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f5bc54e0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc0161ab0 con 0x7f5bc0161550 2024-04-25T01:22:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b20153e70 con 0x7f3b2006d0b0 2024-04-25T01:22:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f577b7fe640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f578006c230 0x7f578006e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f8625d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280ca1e0 0x7f86280c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f8625d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280ca1e0 0x7f86280c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:48944/0 (socket says 172.21.15.77:48944) 2024-04-25T01:22:11.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f8625d9b640 1 -- 172.21.15.77:0/1629112924 learned_addr learned my addr 172.21.15.77:0/1629112924 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f5bbffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bc00b3b10 0x7f5bc00cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f5bbffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bc00b3b10 0x7f5bc00cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56548/0 (socket says 172.21.15.77:56548) 2024-04-25T01:22:11.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f5bbffff640 1 -- 172.21.15.77:0/808367476 learned_addr learned my addr 172.21.15.77:0/808367476 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7f8626d9d640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f862815d0e0 0x7f862815d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7fbe8b7fe640 1 -- 172.21.15.77:0/3170279757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea8068180 con 0x7fbea80b2f80 2024-04-25T01:22:11.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f3b07fff640 1 -- 172.21.15.77:0/2936255665 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18004280 con 0x7f3b2006d0b0 2024-04-25T01:22:11.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f577b7fe640 1 -- 172.21.15.77:0/1577492919 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f578400f040 con 0x7f578c04a390 2024-04-25T01:22:11.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f98e77fe640 1 -- 172.21.15.77:0/3685311099 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99000b9840 con 0x7f98f8004b00 2024-04-25T01:22:11.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f98e77fe640 1 -- 172.21.15.77:0/3685311099 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f99000b9a60 con 0x7f98f8004b00 2024-04-25T01:22:11.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7fb558837640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5540cf520 0x7fb5541585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7fb558837640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55415ad40 0x7fb554154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7fb558837640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb5540cd870 con 0x7fb55406d910 2024-04-25T01:22:11.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f8625d9b640 1 -- 172.21.15.77:0/1629112924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f862815d0e0 msgr2=0x7f862815d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f8625d9b640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f862815d0e0 0x7f862815d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f8625d9b640 1 -- 172.21.15.77:0/1629112924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8610003a00 msgr2=0x7f86280ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f8625d9b640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8610003a00 0x7f86280ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f8625d9b640 1 -- 172.21.15.77:0/1629112924 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86280cd510 con 0x7f86280ca1e0 2024-04-25T01:22:11.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7fb558837640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb55406d3f0 con 0x7fb55415ad40 2024-04-25T01:22:11.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7fb558837640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb5540cd9f0 con 0x7fb5540cf520 2024-04-25T01:22:11.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7fb552d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb55406d910 0x7fb55415b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7fb552d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb55406d910 0x7fb55415b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56536/0 (socket says 172.21.15.77:56536) 2024-04-25T01:22:11.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.676+0000 7fb552d9d640 1 -- 172.21.15.77:0/3412720645 learned_addr learned my addr 172.21.15.77:0/3412720645 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:11.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7fb55259c640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5540cf520 0x7fb5541585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f5792d9d640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f578006c230 0x7f578006e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f5bbffff640 1 -- 172.21.15.77:0/808367476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0161550 msgr2=0x7f5bc00b7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f5bbffff640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0161550 0x7f5bc00b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f5bbffff640 1 -- 172.21.15.77:0/808367476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5bc00ce910 msgr2=0x7f5bc015e8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:22:11.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f5bbffff640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5bc00ce910 0x7f5bc015e8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f5bbffff640 1 -- 172.21.15.77:0/808367476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc015ee10 con 0x7f5bc00b3b10 2024-04-25T01:22:11.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f8625d9b640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280ca1e0 0x7f86280c8230 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f861c00c4f0 tx=0x7f861c00c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f98e77fe640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f98ec06bf60 0x7f98ec06e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7f98e77fe640 1 -- 172.21.15.77:0/3685311099 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f99001278d0 con 0x7f98f8004b00 2024-04-25T01:22:11.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.678+0000 7f98fe7fc640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f98ec06bf60 0x7f98ec06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.678+0000 7f98e57fa640 1 -- 172.21.15.77:0/3685311099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e0005290 con 0x7f98f8004b00 2024-04-25T01:22:11.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.677+0000 7fb55359e640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55415ad40 0x7fb554154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.678+0000 7fb55359e640 1 -- 172.21.15.77:0/3412720645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5540cf520 msgr2=0x7fb5541585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.678+0000 7fb55359e640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5540cf520 0x7fb5541585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.678+0000 7fb55359e640 1 -- 172.21.15.77:0/3412720645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb55406d910 msgr2=0x7fb55415b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:11.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.678+0000 7fb55359e640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb55406d910 0x7fb55415b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:11.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.678+0000 7fb55359e640 1 -- 172.21.15.77:0/3412720645 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb554154db0 con 0x7fb55415ad40 2024-04-25T01:22:11.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.678+0000 7fbe9dffb640 1 -- 172.21.15.77:0/3170279757 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbe980077c0 con 0x7fbea80b2f80 2024-04-25T01:22:11.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.678+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b20068fb0 con 0x7f3b2006d0b0 2024-04-25T01:22:11.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.678+0000 7fbe9dffb640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be80 0x7fbe8c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.679+0000 7f5bbffff640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bc00b3b10 0x7f5bc00cdec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5bb802acc0 tx=0x7f5bb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.679+0000 7fbe9dffb640 1 -- 172.21.15.77:0/3170279757 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbe9808b220 con 0x7fbea80b2f80 2024-04-25T01:22:11.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.679+0000 7fb55359e640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55415ad40 0x7fb554154020 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb544033000 tx=0x7fb544004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.679+0000 7fbeacb4b640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be80 0x7fbe8c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.679+0000 7f98fe7fc640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f98ec06bf60 0x7f98ec06e420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f98f8005f90 tx=0x7f98f0006bd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.679+0000 7f3b07fff640 1 -- 172.21.15.77:0/2936255665 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3b18039070 con 0x7f3b2006d0b0 2024-04-25T01:22:11.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.679+0000 7f5792d9d640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f578006c230 0x7f578006e6f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5788006fd0 tx=0x7f5788008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.679+0000 7fbeacb4b640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be80 0x7fbe8c06e340 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbea4006b10 tx=0x7fbea4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.679+0000 7f3b07fff640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b0c06bf30 0x7f3b0c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.680+0000 7f3b07fff640 1 -- 172.21.15.77:0/2936255665 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3b180abac0 con 0x7f3b2006d0b0 2024-04-25T01:22:11.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.680+0000 7f3b1effd640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b0c06bf30 0x7f3b0c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.680+0000 7f3b1effd640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b0c06bf30 0x7f3b0c06e3f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3b201fb530 tx=0x7f3b10008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.682+0000 7fb54b7fe640 1 -- 172.21.15.77:0/3412720645 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb544042070 con 0x7fb55415ad40 2024-04-25T01:22:11.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.682+0000 7f8616ffd640 1 -- 172.21.15.77:0/1629112924 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c005ce0 con 0x7f86280ca1e0 2024-04-25T01:22:11.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.682+0000 7fb558837640 1 -- 172.21.15.77:0/3412720645 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb55415b800 con 0x7fb55415ad40 2024-04-25T01:22:11.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.682+0000 7f577b7fe640 1 -- 172.21.15.77:0/1577492919 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5784058d10 con 0x7f578c04a390 2024-04-25T01:22:11.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.682+0000 7fb54b7fe640 1 -- 172.21.15.77:0/3412720645 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb544034ba0 con 0x7fb55415ad40 2024-04-25T01:22:11.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.682+0000 7f8616ffd640 1 -- 172.21.15.77:0/1629112924 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f861c007070 con 0x7f86280ca1e0 2024-04-25T01:22:11.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.682+0000 7f862759e640 1 -- 172.21.15.77:0/1629112924 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86280cd7f0 con 0x7f86280ca1e0 2024-04-25T01:22:11.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.682+0000 7f862759e640 1 -- 172.21.15.77:0/1629112924 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86280cd120 con 0x7f86280ca1e0 2024-04-25T01:22:11.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.682+0000 7fb558837640 1 -- 172.21.15.77:0/3412720645 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb55415bdc0 con 0x7fb55415ad40 2024-04-25T01:22:11.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.682+0000 7fb54b7fe640 1 -- 172.21.15.77:0/3412720645 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54403c430 con 0x7fb55415ad40 2024-04-25T01:22:11.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.683+0000 7f8616ffd640 1 -- 172.21.15.77:0/1629112924 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c0074a0 con 0x7f86280ca1e0 2024-04-25T01:22:11.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.684+0000 7f862759e640 1 -- 172.21.15.77:0/1629112924 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f860c005290 con 0x7f86280ca1e0 2024-04-25T01:22:11.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.684+0000 7fb558837640 1 -- 172.21.15.77:0/3412720645 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb554154f80 con 0x7fb55415ad40 2024-04-25T01:22:11.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.684+0000 7fb54b7fe640 1 -- 172.21.15.77:0/3412720645 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb54403c5d0 con 0x7fb55415ad40 2024-04-25T01:22:11.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.684+0000 7f3b07fff640 1 -- 172.21.15.77:0/2936255665 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3b18079b50 con 0x7f3b2006d0b0 2024-04-25T01:22:11.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.684+0000 7fb54b7fe640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb53406be60 0x7fb53406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.684+0000 7fb54b7fe640 1 -- 172.21.15.77:0/3412720645 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb5440abbf0 con 0x7fb55415ad40 2024-04-25T01:22:11.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.684+0000 7fb552d9d640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb53406be60 0x7fb53406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.684+0000 7f8616ffd640 1 -- 172.21.15.77:0/1629112924 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f861c01b400 con 0x7f86280ca1e0 2024-04-25T01:22:11.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.685+0000 7fb552d9d640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb53406be60 0x7fb53406e320 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb55415e480 tx=0x7fb54c009110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.685+0000 7f8616ffd640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860806bf00 0x7f860806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.685+0000 7f862659c640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860806bf00 0x7f860806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.685+0000 7f8616ffd640 1 -- 172.21.15.77:0/1629112924 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f861c08adc0 con 0x7f86280ca1e0 2024-04-25T01:22:11.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.686+0000 7f862659c640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860806bf00 0x7f860806e3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8618033000 tx=0x7f861802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.689+0000 7f98e77fe640 1 -- 172.21.15.77:0/3685311099 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f99000f5e90 con 0x7f98f8004b00 2024-04-25T01:22:11.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.689+0000 7fb54b7fe640 1 -- 172.21.15.77:0/3412720645 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb544079d30 con 0x7fb55415ad40 2024-04-25T01:22:11.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.693+0000 7f8616ffd640 1 -- 172.21.15.77:0/1629112924 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f861c0594a0 con 0x7f86280ca1e0 2024-04-25T01:22:11.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.697+0000 7fbe9dffb640 1 -- 172.21.15.77:0/3170279757 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbe980592b0 con 0x7fbea80b2f80 2024-04-25T01:22:11.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.698+0000 7f5bbcff9640 1 -- 172.21.15.77:0/808367476 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8041030 con 0x7f5bc00b3b10 2024-04-25T01:22:11.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.698+0000 7f5bbcff9640 1 -- 172.21.15.77:0/808367476 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb8037040 con 0x7f5bc00b3b10 2024-04-25T01:22:11.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.698+0000 7f5bbcff9640 1 -- 172.21.15.77:0/808367476 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8036070 con 0x7f5bc00b3b10 2024-04-25T01:22:11.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.698+0000 7f5bc54e0640 1 -- 172.21.15.77:0/808367476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc00b8190 con 0x7f5bc00b3b10 2024-04-25T01:22:11.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.698+0000 7f5bc54e0640 1 -- 172.21.15.77:0/808367476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc00b86f0 con 0x7f5bc00b3b10 2024-04-25T01:22:11.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.701+0000 7f5bbcff9640 1 -- 172.21.15.77:0/808367476 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5bb8044070 con 0x7f5bc00b3b10 2024-04-25T01:22:11.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.701+0000 7f5bbcff9640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5bac06bf00 0x7f5bac06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:11.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.701+0000 7f5bbcff9640 1 -- 172.21.15.77:0/808367476 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5bb80abad0 con 0x7f5bc00b3b10 2024-04-25T01:22:11.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.703+0000 7f5bbf7fe640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5bac06bf00 0x7f5bac06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:11.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.704+0000 7f5bbf7fe640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5bac06bf00 0x7f5bac06e3c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00d2670 tx=0x7f5bb4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:11.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.704+0000 7f5bc54e0640 1 -- 172.21.15.77:0/808367476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ba0005290 con 0x7f5bc00b3b10 2024-04-25T01:22:11.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:11.714+0000 7f5bbcff9640 1 -- 172.21.15.77:0/808367476 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5bb804b070 con 0x7f5bc00b3b10 2024-04-25T01:22:12.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.021+0000 7f3796b70640 1 -- 172.21.15.77:0/3311640774 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f3770005580 con 0x7f3788053a90 2024-04-25T01:22:12.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.022+0000 7f378d7fa640 1 -- 172.21.15.77:0/3311640774 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3784079850 con 0x7f3788053a90 2024-04-25T01:22:12.003 INFO:teuthology.orchestra.run.smithi077.stdout:55834574859 2024-04-25T01:22:12.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.025+0000 7f377affd640 1 -- 172.21.15.77:0/3311640774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f377406bf00 msgr2=0x7f377406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.026+0000 7f377affd640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f377406bf00 0x7f377406e3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f37880ec070 tx=0x7f377c008040 comp rx=0 tx=0).stop 2024-04-25T01:22:12.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.026+0000 7f377affd640 1 -- 172.21.15.77:0/3311640774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3788053a90 msgr2=0x7f37880d1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.026+0000 7f377affd640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3788053a90 0x7f37880d1ce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3784009b90 tx=0x7f37840046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.026+0000 7f377affd640 1 -- 172.21.15.77:0/3311640774 shutdown_connections 2024-04-25T01:22:12.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.026+0000 7f377affd640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f377406bf00 0x7f377406e3c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.026+0000 7f377affd640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3788053e70 0x7f37880e2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.026+0000 7f377affd640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3788054980 0x7f37880caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.027+0000 7f377affd640 1 --2- 172.21.15.77:0/3311640774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3788053a90 0x7f37880d1ce0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.027+0000 7f377affd640 1 -- 172.21.15.77:0/3311640774 >> 172.21.15.77:0/3311640774 conn(0x7f3788055c50 msgr2=0x7f37880561f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.027+0000 7f377affd640 1 -- 172.21.15.77:0/3311640774 shutdown_connections 2024-04-25T01:22:12.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.027+0000 7f377affd640 1 -- 172.21.15.77:0/3311640774 wait complete. 2024-04-25T01:22:12.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.2 2024-04-25T01:22:12.024 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:12.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.076+0000 7f5bc54e0640 1 -- 172.21.15.77:0/808367476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f5ba0005580 con 0x7f5bc00b3b10 2024-04-25T01:22:12.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.077+0000 7f5bbcff9640 1 -- 172.21.15.77:0/808367476 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5bc00b8190 con 0x7f5bc00b3b10 2024-04-25T01:22:12.058 INFO:teuthology.orchestra.run.smithi077.stdout:55834574858 2024-04-25T01:22:12.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.079+0000 7f5bc54e0640 1 -- 172.21.15.77:0/808367476 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5bac06bf00 msgr2=0x7f5bac06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.079+0000 7f5bc54e0640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5bac06bf00 0x7f5bac06e3c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00d2670 tx=0x7f5bb4009190 comp rx=0 tx=0).stop 2024-04-25T01:22:12.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.079+0000 7f5bc54e0640 1 -- 172.21.15.77:0/808367476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bc00b3b10 msgr2=0x7f5bc00cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.079+0000 7f5bc54e0640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bc00b3b10 0x7f5bc00cdec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5bb802acc0 tx=0x7f5bb8004a40 comp rx=0 tx=0).stop 2024-04-25T01:22:12.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.079+0000 7f5bc54e0640 1 -- 172.21.15.77:0/808367476 shutdown_connections 2024-04-25T01:22:12.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.079+0000 7f5bc54e0640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5bac06bf00 0x7f5bac06e3c0 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00d2670 tx=0x7f5bb4009190 comp rx=0 tx=0).stop 2024-04-25T01:22:12.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.079+0000 7f5bc54e0640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0161550 0x7f5bc00b7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.079+0000 7f5bc54e0640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5bc00ce910 0x7f5bc015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.079+0000 7f5bc54e0640 1 --2- 172.21.15.77:0/808367476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bc00b3b10 0x7f5bc00cdec0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.079+0000 7f5bc54e0640 1 -- 172.21.15.77:0/808367476 >> 172.21.15.77:0/808367476 conn(0x7f5bc009d2b0 msgr2=0x7f5bc0160680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.080+0000 7f5bc54e0640 1 -- 172.21.15.77:0/808367476 shutdown_connections 2024-04-25T01:22:12.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.080+0000 7f5bc54e0640 1 -- 172.21.15.77:0/808367476 wait complete. 2024-04-25T01:22:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.080+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f578c00c9e0 con 0x7f578c04a390 2024-04-25T01:22:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.081+0000 7f577b7fe640 1 -- 172.21.15.77:0/1577492919 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5784058d10 con 0x7f578c04a390 2024-04-25T01:22:12.062 INFO:teuthology.orchestra.run.smithi077.stdout:55834574860 2024-04-25T01:22:12.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.084+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f578006c230 msgr2=0x7f578006e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.084+0000 7f5798829640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f578006c230 0x7f578006e6f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5788006fd0 tx=0x7f5788008040 comp rx=0 tx=0).stop 2024-04-25T01:22:12.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.084+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f578c04a390 msgr2=0x7f578c050aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.084+0000 7f5798829640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f578c04a390 0x7f578c050aa0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f578400ebe0 tx=0x7f578400c520 comp rx=0 tx=0).stop 2024-04-25T01:22:12.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.085+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 shutdown_connections 2024-04-25T01:22:12.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.085+0000 7f5798829640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f578006c230 0x7f578006e6f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.085+0000 7f5798829640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f578c057480 0x7f578c035e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.085+0000 7f5798829640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f578c04a390 0x7f578c050aa0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.085+0000 7f5798829640 1 --2- 172.21.15.77:0/1577492919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f578c049fb0 0x7f578c0da2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.085+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 >> 172.21.15.77:0/1577492919 conn(0x7f578c020040 msgr2=0x7f578c0e7290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.085+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 shutdown_connections 2024-04-25T01:22:12.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.085+0000 7f5798829640 1 -- 172.21.15.77:0/1577492919 wait complete. 2024-04-25T01:22:12.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.088+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f3b2015b3c0 con 0x7f3b2006d0b0 2024-04-25T01:22:12.070 INFO:teuthology.orchestra.run.smithi077.stdout:55834574858 2024-04-25T01:22:12.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.089+0000 7f3b07fff640 1 -- 172.21.15.77:0/2936255665 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3b1807de50 con 0x7f3b2006d0b0 2024-04-25T01:22:12.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b0c06bf30 msgr2=0x7f3b0c06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b0c06bf30 0x7f3b0c06e3f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3b201fb530 tx=0x7f3b10008040 comp rx=0 tx=0).stop 2024-04-25T01:22:12.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b2006d0b0 msgr2=0x7f3b201566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b2006d0b0 0x7f3b201566b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3b18035000 tx=0x7f3b18004a40 comp rx=0 tx=0).stop 2024-04-25T01:22:12.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 shutdown_connections 2024-04-25T01:22:12.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b0c06bf30 0x7f3b0c06e3f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b200aee90 0x7f3b2014cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b20157930 0x7f3b2014d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 --2- 172.21.15.77:0/2936255665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b2006d0b0 0x7f3b201566b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 >> 172.21.15.77:0/2936255665 conn(0x7f3b2006b7e0 msgr2=0x7f3b200b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 shutdown_connections 2024-04-25T01:22:12.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.092+0000 7f3b24eff640 1 -- 172.21.15.77:0/2936255665 wait complete. 2024-04-25T01:22:12.075 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.4 2024-04-25T01:22:12.075 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:12.084 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.5 2024-04-25T01:22:12.084 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:12.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.103+0000 7fb558837640 1 -- 172.21.15.77:0/3412720645 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb554075350 con 0x7fb55415ad40 2024-04-25T01:22:12.085 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.0 2024-04-25T01:22:12.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.104+0000 7fb54b7fe640 1 -- 172.21.15.77:0/3412720645 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb54407e030 con 0x7fb55415ad40 2024-04-25T01:22:12.085 INFO:teuthology.orchestra.run.smithi077.stdout:55834574860 2024-04-25T01:22:12.086 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:12.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.104+0000 7fbe8b7fe640 1 -- 172.21.15.77:0/3170279757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fbea80d3a70 con 0x7fbea80b2f80 2024-04-25T01:22:12.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.105+0000 7fbe9dffb640 1 -- 172.21.15.77:0/3170279757 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbe9805d5b0 con 0x7fbea80b2f80 2024-04-25T01:22:12.086 INFO:teuthology.orchestra.run.smithi077.stdout:55834574860 2024-04-25T01:22:12.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.106+0000 7f862759e640 1 -- 172.21.15.77:0/1629112924 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f860c005580 con 0x7f86280ca1e0 2024-04-25T01:22:12.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7f8616ffd640 1 -- 172.21.15.77:0/1629112924 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f861c058da0 con 0x7f86280ca1e0 2024-04-25T01:22:12.088 INFO:teuthology.orchestra.run.smithi077.stdout:55834574858 2024-04-25T01:22:12.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fbe8b7fe640 1 -- 172.21.15.77:0/3170279757 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be80 msgr2=0x7fbe8c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fbe8b7fe640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be80 0x7fbe8c06e340 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbea4006b10 tx=0x7fbea4008040 comp rx=0 tx=0).stop 2024-04-25T01:22:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 -- 172.21.15.77:0/3412720645 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb53406be60 msgr2=0x7fb53406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb53406be60 0x7fb53406e320 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb55415e480 tx=0x7fb54c009110 comp rx=0 tx=0).stop 2024-04-25T01:22:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 -- 172.21.15.77:0/3412720645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55415ad40 msgr2=0x7fb554154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55415ad40 0x7fb554154020 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb544033000 tx=0x7fb544004a40 comp rx=0 tx=0).stop 2024-04-25T01:22:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 -- 172.21.15.77:0/3412720645 shutdown_connections 2024-04-25T01:22:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb53406be60 0x7fb53406e320 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55415ad40 0x7fb554154020 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5540cf520 0x7fb5541585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 --2- 172.21.15.77:0/3412720645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb55406d910 0x7fb55415b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 -- 172.21.15.77:0/3412720645 >> 172.21.15.77:0/3412720645 conn(0x7fb5540b1660 msgr2=0x7fb554071390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fbe8b7fe640 1 -- 172.21.15.77:0/3170279757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea80b2f80 msgr2=0x7fbea81ead00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fbe8b7fe640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea80b2f80 0x7fbea81ead00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbe9800c850 tx=0x7fbe9800cd10 comp rx=0 tx=0).stop 2024-04-25T01:22:12.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fbe8b7fe640 1 -- 172.21.15.77:0/3170279757 shutdown_connections 2024-04-25T01:22:12.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 -- 172.21.15.77:0/3412720645 shutdown_connections 2024-04-25T01:22:12.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fbe8b7fe640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be80 0x7fbe8c06e340 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fbe8b7fe640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbea80b7ad0 0x7fbea81ea550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fbe8b7fe640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea80b34f0 0x7fbea80b6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fbe8b7fe640 1 --2- 172.21.15.77:0/3170279757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea80b2f80 0x7fbea81ead00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fb558837640 1 -- 172.21.15.77:0/3412720645 wait complete. 2024-04-25T01:22:12.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.107+0000 7fbe8b7fe640 1 -- 172.21.15.77:0/3170279757 >> 172.21.15.77:0/3170279757 conn(0x7fbea80d09f0 msgr2=0x7fbea806c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.108+0000 7fbe8b7fe640 1 -- 172.21.15.77:0/3170279757 shutdown_connections 2024-04-25T01:22:12.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.108+0000 7fbe8b7fe640 1 -- 172.21.15.77:0/3170279757 wait complete. 2024-04-25T01:22:12.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f98e57fa640 1 -- 172.21.15.77:0/3685311099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f98e0005580 con 0x7f98f8004b00 2024-04-25T01:22:12.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 -- 172.21.15.77:0/1629112924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860806bf00 msgr2=0x7f860806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860806bf00 0x7f860806e3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8618033000 tx=0x7f861802e000 comp rx=0 tx=0).stop 2024-04-25T01:22:12.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 -- 172.21.15.77:0/1629112924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280ca1e0 msgr2=0x7f86280c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280ca1e0 0x7f86280c8230 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f861c00c4f0 tx=0x7f861c00c9b0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f98e77fe640 1 -- 172.21.15.77:0/3685311099 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f99000f5790 con 0x7f98f8004b00 2024-04-25T01:22:12.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 -- 172.21.15.77:0/1629112924 shutdown_connections 2024-04-25T01:22:12.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860806bf00 0x7f860806e3c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f862815d0e0 0x7f862815d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280ca1e0 0x7f86280c8230 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 --2- 172.21.15.77:0/1629112924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8610003a00 0x7f86280ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 -- 172.21.15.77:0/1629112924 >> 172.21.15.77:0/1629112924 conn(0x7f862809d170 msgr2=0x7f86280b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.094 INFO:teuthology.orchestra.run.smithi077.stdout:55834574859 2024-04-25T01:22:12.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 -- 172.21.15.77:0/1629112924 shutdown_connections 2024-04-25T01:22:12.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.110+0000 7f862759e640 1 -- 172.21.15.77:0/1629112924 wait complete. 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.113+0000 7f98e57fa640 1 -- 172.21.15.77:0/3685311099 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f98ec06bf60 msgr2=0x7f98ec06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.114+0000 7f98e57fa640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f98ec06bf60 0x7f98ec06e420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f98f8005f90 tx=0x7f98f0006bd0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.114+0000 7f98e57fa640 1 -- 172.21.15.77:0/3685311099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f8004b00 msgr2=0x7f98f804ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.114+0000 7f98e57fa640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f8004b00 0x7f98f804ea20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f990006ac00 tx=0x7f990006b050 comp rx=0 tx=0).stop 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.114+0000 7f98e57fa640 1 -- 172.21.15.77:0/3685311099 shutdown_connections 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.114+0000 7f98e57fa640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f98ec06bf60 0x7f98ec06e420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.114+0000 7f98e57fa640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98f804c9f0 0x7f98f804a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.114+0000 7f98e57fa640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98f80e2d80 0x7f98f804d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.114+0000 7f98e57fa640 1 --2- 172.21.15.77:0/3685311099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f8004b00 0x7f98f804ea20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.114+0000 7f98e57fa640 1 -- 172.21.15.77:0/3685311099 >> 172.21.15.77:0/3685311099 conn(0x7f98f800a400 msgr2=0x7f98f804fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.114+0000 7f98e57fa640 1 -- 172.21.15.77:0/3685311099 shutdown_connections 2024-04-25T01:22:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.114+0000 7f98e57fa640 1 -- 172.21.15.77:0/3685311099 wait complete. 2024-04-25T01:22:12.102 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.6 2024-04-25T01:22:12.102 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:12.104 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2024-04-25T01:22:12.105 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:12.106 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.3 2024-04-25T01:22:12.106 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:12.110 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.1 2024-04-25T01:22:12.110 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:12.110 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T01:22:12.110 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:22:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.266+0000 7f290b961640 1 Processor -- start 2024-04-25T01:22:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.266+0000 7f290b961640 1 -- start start 2024-04-25T01:22:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.267+0000 7f290b961640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f290406fcd0 0x7f290415ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.267+0000 7f290b961640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f29040ade10 0x7f290406c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.267+0000 7f290b961640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f290406cac0 con 0x7f290406c670 2024-04-25T01:22:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.267+0000 7f290b961640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f29041589e0 con 0x7f29040ade10 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.267+0000 7f290b961640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f29040b3890 con 0x7f290406fcd0 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.267+0000 7f2908efc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f29040ade10 0x7f290406c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.267+0000 7f2909efe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f290406c670 0x7f2904161ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56550/0 (socket says 172.21.15.77:56550) 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.267+0000 7f2909efe640 1 -- 172.21.15.77:0/2212757714 learned_addr learned my addr 172.21.15.77:0/2212757714 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.268+0000 7f29096fd640 1 --2- 172.21.15.77:0/2212757714 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f290406fcd0 0x7f290415ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.268+0000 7f2908efc640 1 -- 172.21.15.77:0/2212757714 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f290406fcd0 msgr2=0x7f290415ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.268+0000 7f2908efc640 1 --2- 172.21.15.77:0/2212757714 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f290406fcd0 0x7f290415ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.268+0000 7f2908efc640 1 -- 172.21.15.77:0/2212757714 >> v1:172.21.15.77:6789/0 conn(0x7f290406c670 legacy=0x7f2904161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.268+0000 7f28fbfff640 1 -- 172.21.15.77:0/2212757714 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4159472807 0 0) 0x7f290406cac0 con 0x7f290406c670 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.268+0000 7f2908efc640 1 -- 172.21.15.77:0/2212757714 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29040ae290 con 0x7f29040ade10 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.268+0000 7f29096fd640 1 --2- 172.21.15.77:0/2212757714 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f290406fcd0 0x7f290415ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.268+0000 7f2908efc640 1 --2- 172.21.15.77:0/2212757714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29040ade10 0x7f290406c130 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f28f40097a0 tx=0x7f28f402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4106503306de910a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.269+0000 7f28fbfff640 1 -- 172.21.15.77:0/2212757714 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4037070 con 0x7f29040ade10 2024-04-25T01:22:12.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.269+0000 7f28fbfff640 1 -- 172.21.15.77:0/2212757714 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28f402ae50 con 0x7f29040ade10 2024-04-25T01:22:12.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.269+0000 7f28fbfff640 1 -- 172.21.15.77:0/2212757714 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4032660 con 0x7f29040ade10 2024-04-25T01:22:12.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.269+0000 7f290b961640 1 -- 172.21.15.77:0/2212757714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29040ade10 msgr2=0x7f290406c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.269+0000 7f290b961640 1 --2- 172.21.15.77:0/2212757714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29040ade10 0x7f290406c130 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f28f40097a0 tx=0x7f28f402a100 comp rx=0 tx=0).stop 2024-04-25T01:22:12.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.269+0000 7f290b961640 1 -- 172.21.15.77:0/2212757714 shutdown_connections 2024-04-25T01:22:12.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.269+0000 7f290b961640 1 --2- 172.21.15.77:0/2212757714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29040ade10 0x7f290406c130 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.269+0000 7f290b961640 1 --2- 172.21.15.77:0/2212757714 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f290406fcd0 0x7f290415ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.269+0000 7f290b961640 1 -- 172.21.15.77:0/2212757714 >> 172.21.15.77:0/2212757714 conn(0x7f2904071550 msgr2=0x7f2904071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.269+0000 7f290b961640 1 -- 172.21.15.77:0/2212757714 shutdown_connections 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.269+0000 7f290b961640 1 -- 172.21.15.77:0/2212757714 wait complete. 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.270+0000 7f290b961640 1 Processor -- start 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.270+0000 7f290b961640 1 -- start start 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f290b961640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f290406c670 0x7f29040cf720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f290b961640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f290406fcd0 0x7f2904143e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f290b961640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29040ade10 0x7f2904158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f290b961640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f29040ae7f0 con 0x7f290406c670 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f290b961640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f29040b3560 con 0x7f29040ade10 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f290b961640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f29040aead0 con 0x7f290406fcd0 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f29096fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f290406c670 0x7f29040cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f29096fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f290406c670 0x7f29040cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56564/0 (socket says 172.21.15.77:56564) 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f29096fd640 1 -- 172.21.15.77:0/4064935835 learned_addr learned my addr 172.21.15.77:0/4064935835 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f2908efc640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f290406fcd0 0x7f2904143e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f2909efe640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29040ade10 0x7f2904158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f29096fd640 1 -- 172.21.15.77:0/4064935835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f290406fcd0 msgr2=0x7f2904143e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f29096fd640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f290406fcd0 0x7f2904143e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f29096fd640 1 -- 172.21.15.77:0/4064935835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29040ade10 msgr2=0x7f2904158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f29096fd640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29040ade10 0x7f2904158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f29096fd640 1 -- 172.21.15.77:0/4064935835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29040cce40 con 0x7f290406c670 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.271+0000 7f2909efe640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29040ade10 0x7f2904158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.272+0000 7f29096fd640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f290406c670 0x7f29040cf720 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f290000c7f0 tx=0x7f290000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.272+0000 7f28f9ffb640 1 -- 172.21.15.77:0/4064935835 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900015070 con 0x7f290406c670 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.272+0000 7f28f9ffb640 1 -- 172.21.15.77:0/4064935835 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2900007a30 con 0x7f290406c670 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.272+0000 7f28f9ffb640 1 -- 172.21.15.77:0/4064935835 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900007e00 con 0x7f290406c670 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.272+0000 7f290b961640 1 -- 172.21.15.77:0/4064935835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29040cd120 con 0x7f290406c670 2024-04-25T01:22:12.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.272+0000 7f290b961640 1 -- 172.21.15.77:0/4064935835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29040cd6e0 con 0x7f290406c670 2024-04-25T01:22:12.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.273+0000 7f290b961640 1 -- 172.21.15.77:0/4064935835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29040b3250 con 0x7f290406c670 2024-04-25T01:22:12.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.273+0000 7f28f9ffb640 1 -- 172.21.15.77:0/4064935835 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2900018070 con 0x7f290406c670 2024-04-25T01:22:12.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.274+0000 7f28f9ffb640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28e806be30 0x7f28e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.274+0000 7f28f9ffb640 1 -- 172.21.15.77:0/4064935835 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f290008b190 con 0x7f290406c670 2024-04-25T01:22:12.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.274+0000 7f2908efc640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28e806be30 0x7f28e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.274+0000 7f2908efc640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28e806be30 0x7f28e806e2f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f28f4034000 tx=0x7f28f402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:12.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.278+0000 7f28f9ffb640 1 -- 172.21.15.77:0/4064935835 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f29040b3250 con 0x7f290406c670 2024-04-25T01:22:12.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.446+0000 7f290b961640 1 -- 172.21.15.77:0/4064935835 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2904068fb0 con 0x7f28e806be30 2024-04-25T01:22:12.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.454+0000 7f28f9ffb640 1 -- 172.21.15.77:0/4064935835 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7f2904068fb0 con 0x7f28e806be30 2024-04-25T01:22:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:12.440 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:22:12.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 -- 172.21.15.77:0/4064935835 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28e806be30 msgr2=0x7f28e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28e806be30 0x7f28e806e2f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f28f4034000 tx=0x7f28f402e000 comp rx=0 tx=0).stop 2024-04-25T01:22:12.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 -- 172.21.15.77:0/4064935835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f290406c670 msgr2=0x7f29040cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f290406c670 0x7f29040cf720 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f290000c7f0 tx=0x7f290000cbf0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 -- 172.21.15.77:0/4064935835 shutdown_connections 2024-04-25T01:22:12.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28e806be30 0x7f28e806e2f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29040ade10 0x7f2904158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f290406fcd0 0x7f2904143e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 --2- 172.21.15.77:0/4064935835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f290406c670 0x7f29040cf720 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 -- 172.21.15.77:0/4064935835 >> 172.21.15.77:0/4064935835 conn(0x7f2904071550 msgr2=0x7f290406d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 -- 172.21.15.77:0/4064935835 shutdown_connections 2024-04-25T01:22:12.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.457+0000 7f290b961640 1 -- 172.21.15.77:0/4064935835 wait complete. 2024-04-25T01:22:12.449 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-25T01:22:11.161505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219400,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755320,"statfs":{"total":773094113280,"available":772869447680,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003202"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471974+0000","last_change":"2024-04-25T01:21:45.082729+0000","last_active":"2024-04-25T01:21:51.471974+0000","last_peered":"2024-04-25T01:21:51.471974+0000","last_clean":"2024-04-25T01:21:51.471974+0000","last_became_active":"2024-04-25T01:21:45.082441+0000","last_became_peered":"2024-04-25T01:21:45.082441+0000","last_unstale":"2024-04-25T01:21:51.471974+0000","last_undegraded":"2024-04-25T01:21:51.471974+0000","last_fullsized":"2024-04-25T01:21:51.471974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:31.228997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472236+0000","last_change":"2024-04-25T01:21:46.093885+0000","last_active":"2024-04-25T01:21:51.472236+0000","last_peered":"2024-04-25T01:21:51.472236+0000","last_clean":"2024-04-25T01:21:51.472236+0000","last_became_active":"2024-04-25T01:21:46.092142+0000","last_became_peered":"2024-04-25T01:21:46.092142+0000","last_unstale":"2024-04-25T01:21:51.472236+0000","last_undegraded":"2024-04-25T01:21:51.472236+0000","last_fullsized":"2024-04-25T01:21:51.472236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471738+0000","last_change":"2024-04-25T01:21:50.187217+0000","last_active":"2024-04-25T01:21:51.471738+0000","last_peered":"2024-04-25T01:21:51.471738+0000","last_clean":"2024-04-25T01:21:51.471738+0000","last_became_active":"2024-04-25T01:21:50.186698+0000","last_became_peered":"2024-04-25T01:21:50.186698+0000","last_unstale":"2024-04-25T01:21:51.471738+0000","last_undegraded":"2024-04-25T01:21:51.471738+0000","last_fullsized":"2024-04-25T01:21:51.471738+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:50.627648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472373+0000","last_change":"2024-04-25T01:21:45.078191+0000","last_active":"2024-04-25T01:21:51.472373+0000","last_peered":"2024-04-25T01:21:51.472373+0000","last_clean":"2024-04-25T01:21:51.472373+0000","last_became_active":"2024-04-25T01:21:45.077949+0000","last_became_peered":"2024-04-25T01:21:45.077949+0000","last_unstale":"2024-04-25T01:21:51.472373+0000","last_undegraded":"2024-04-25T01:21:51.472373+0000","last_fullsized":"2024-04-25T01:21:51.472373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:23.371789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:46.091354+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:46.090827+0000","last_became_peered":"2024-04-25T01:21:46.090827+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.117971+0000","last_change":"2024-04-25T01:21:50.126246+0000","last_active":"2024-04-25T01:21:51.117971+0000","last_peered":"2024-04-25T01:21:51.117971+0000","last_clean":"2024-04-25T01:21:51.117971+0000","last_became_active":"2024-04-25T01:21:50.125823+0000","last_became_peered":"2024-04-25T01:21:50.125823+0000","last_unstale":"2024-04-25T01:21:51.117971+0000","last_undegraded":"2024-04-25T01:21:51.117971+0000","last_fullsized":"2024-04-25T01:21:51.117971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:05.469540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472877+0000","last_change":"2024-04-25T01:21:45.080811+0000","last_active":"2024-04-25T01:21:51.472877+0000","last_peered":"2024-04-25T01:21:51.472877+0000","last_clean":"2024-04-25T01:21:51.472877+0000","last_became_active":"2024-04-25T01:21:45.079902+0000","last_became_peered":"2024-04-25T01:21:45.079902+0000","last_unstale":"2024-04-25T01:21:51.472877+0000","last_undegraded":"2024-04-25T01:21:51.472877+0000","last_fullsized":"2024-04-25T01:21:51.472877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:15.208981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472902+0000","last_change":"2024-04-25T01:21:46.090986+0000","last_active":"2024-04-25T01:21:51.472902+0000","last_peered":"2024-04-25T01:21:51.472902+0000","last_clean":"2024-04-25T01:21:51.472902+0000","last_became_active":"2024-04-25T01:21:46.090715+0000","last_became_peered":"2024-04-25T01:21:46.090715+0000","last_unstale":"2024-04-25T01:21:51.472902+0000","last_undegraded":"2024-04-25T01:21:51.472902+0000","last_fullsized":"2024-04-25T01:21:51.472902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:27.583834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118940+0000","last_change":"2024-04-25T01:21:50.128507+0000","last_active":"2024-04-25T01:21:51.118940+0000","last_peered":"2024-04-25T01:21:51.118940+0000","last_clean":"2024-04-25T01:21:51.118940+0000","last_became_active":"2024-04-25T01:21:50.128116+0000","last_became_peered":"2024-04-25T01:21:50.128116+0000","last_unstale":"2024-04-25T01:21:51.118940+0000","last_undegraded":"2024-04-25T01:21:51.118940+0000","last_fullsized":"2024-04-25T01:21:51.118940+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.077404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472303+0000","last_change":"2024-04-25T01:21:45.080401+0000","last_active":"2024-04-25T01:21:51.472303+0000","last_peered":"2024-04-25T01:21:51.472303+0000","last_clean":"2024-04-25T01:21:51.472303+0000","last_became_active":"2024-04-25T01:21:45.079961+0000","last_became_peered":"2024-04-25T01:21:45.079961+0000","last_unstale":"2024-04-25T01:21:51.472303+0000","last_undegraded":"2024-04-25T01:21:51.472303+0000","last_fullsized":"2024-04-25T01:21:51.472303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473268+0000","last_change":"2024-04-25T01:21:46.090524+0000","last_active":"2024-04-25T01:21:51.473268+0000","last_peered":"2024-04-25T01:21:51.473268+0000","last_clean":"2024-04-25T01:21:51.473268+0000","last_became_active":"2024-04-25T01:21:46.089869+0000","last_became_peered":"2024-04-25T01:21:46.089869+0000","last_unstale":"2024-04-25T01:21:51.473268+0000","last_undegraded":"2024-04-25T01:21:51.473268+0000","last_fullsized":"2024-04-25T01:21:51.473268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:24.344315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118039+0000","last_change":"2024-04-25T01:21:50.118075+0000","last_active":"2024-04-25T01:21:51.118039+0000","last_peered":"2024-04-25T01:21:51.118039+0000","last_clean":"2024-04-25T01:21:51.118039+0000","last_became_active":"2024-04-25T01:21:50.117657+0000","last_became_peered":"2024-04-25T01:21:50.117657+0000","last_unstale":"2024-04-25T01:21:51.118039+0000","last_undegraded":"2024-04-25T01:21:51.118039+0000","last_fullsized":"2024-04-25T01:21:51.118039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:29.510529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118231+0000","last_change":"2024-04-25T01:21:46.093011+0000","last_active":"2024-04-25T01:21:51.118231+0000","last_peered":"2024-04-25T01:21:51.118231+0000","last_clean":"2024-04-25T01:21:51.118231+0000","last_became_active":"2024-04-25T01:21:46.092710+0000","last_became_peered":"2024-04-25T01:21:46.092710+0000","last_unstale":"2024-04-25T01:21:51.118231+0000","last_undegraded":"2024-04-25T01:21:51.118231+0000","last_fullsized":"2024-04-25T01:21:51.118231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472603+0000","last_change":"2024-04-25T01:21:45.080010+0000","last_active":"2024-04-25T01:21:51.472603+0000","last_peered":"2024-04-25T01:21:51.472603+0000","last_clean":"2024-04-25T01:21:51.472603+0000","last_became_active":"2024-04-25T01:21:45.079732+0000","last_became_peered":"2024-04-25T01:21:45.079732+0000","last_unstale":"2024-04-25T01:21:51.472603+0000","last_undegraded":"2024-04-25T01:21:51.472603+0000","last_fullsized":"2024-04-25T01:21:51.472603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119330+0000","last_change":"2024-04-25T01:21:50.121929+0000","last_active":"2024-04-25T01:21:51.119330+0000","last_peered":"2024-04-25T01:21:51.119330+0000","last_clean":"2024-04-25T01:21:51.119330+0000","last_became_active":"2024-04-25T01:21:50.121288+0000","last_became_peered":"2024-04-25T01:21:50.121288+0000","last_unstale":"2024-04-25T01:21:51.119330+0000","last_undegraded":"2024-04-25T01:21:51.119330+0000","last_fullsized":"2024-04-25T01:21:51.119330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473407+0000","last_change":"2024-04-25T01:21:46.082691+0000","last_active":"2024-04-25T01:21:51.473407+0000","last_peered":"2024-04-25T01:21:51.473407+0000","last_clean":"2024-04-25T01:21:51.473407+0000","last_became_active":"2024-04-25T01:21:46.082266+0000","last_became_peered":"2024-04-25T01:21:46.082266+0000","last_unstale":"2024-04-25T01:21:51.473407+0000","last_undegraded":"2024-04-25T01:21:51.473407+0000","last_fullsized":"2024-04-25T01:21:51.473407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:22.971333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473925+0000","last_change":"2024-04-25T01:21:45.076031+0000","last_active":"2024-04-25T01:21:51.473925+0000","last_peered":"2024-04-25T01:21:51.473925+0000","last_clean":"2024-04-25T01:21:51.473925+0000","last_became_active":"2024-04-25T01:21:45.075769+0000","last_became_peered":"2024-04-25T01:21:45.075769+0000","last_unstale":"2024-04-25T01:21:51.473925+0000","last_undegraded":"2024-04-25T01:21:51.473925+0000","last_fullsized":"2024-04-25T01:21:51.473925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118699+0000","last_change":"2024-04-25T01:21:50.122266+0000","last_active":"2024-04-25T01:21:51.118699+0000","last_peered":"2024-04-25T01:21:51.118699+0000","last_clean":"2024-04-25T01:21:51.118699+0000","last_became_active":"2024-04-25T01:21:50.121879+0000","last_became_peered":"2024-04-25T01:21:50.121879+0000","last_unstale":"2024-04-25T01:21:51.118699+0000","last_undegraded":"2024-04-25T01:21:51.118699+0000","last_fullsized":"2024-04-25T01:21:51.118699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:51.825101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471956+0000","last_change":"2024-04-25T01:21:46.093175+0000","last_active":"2024-04-25T01:21:51.471956+0000","last_peered":"2024-04-25T01:21:51.471956+0000","last_clean":"2024-04-25T01:21:51.471956+0000","last_became_active":"2024-04-25T01:21:46.092327+0000","last_became_peered":"2024-04-25T01:21:46.092327+0000","last_unstale":"2024-04-25T01:21:51.471956+0000","last_undegraded":"2024-04-25T01:21:51.471956+0000","last_fullsized":"2024-04-25T01:21:51.471956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473162+0000","last_change":"2024-04-25T01:21:45.077148+0000","last_active":"2024-04-25T01:21:51.473162+0000","last_peered":"2024-04-25T01:21:51.473162+0000","last_clean":"2024-04-25T01:21:51.473162+0000","last_became_active":"2024-04-25T01:21:45.076858+0000","last_became_peered":"2024-04-25T01:21:45.076858+0000","last_unstale":"2024-04-25T01:21:51.473162+0000","last_undegraded":"2024-04-25T01:21:51.473162+0000","last_fullsized":"2024-04-25T01:21:51.473162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:15.208981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473795+0000","last_change":"2024-04-25T01:21:50.182249+0000","last_active":"2024-04-25T01:21:51.473795+0000","last_peered":"2024-04-25T01:21:51.473795+0000","last_clean":"2024-04-25T01:21:51.473795+0000","last_became_active":"2024-04-25T01:21:50.181860+0000","last_became_peered":"2024-04-25T01:21:50.181860+0000","last_unstale":"2024-04-25T01:21:51.473795+0000","last_undegraded":"2024-04-25T01:21:51.473795+0000","last_fullsized":"2024-04-25T01:21:51.473795+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:38.806941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119159+0000","last_change":"2024-04-25T01:21:46.085766+0000","last_active":"2024-04-25T01:21:51.119159+0000","last_peered":"2024-04-25T01:21:51.119159+0000","last_clean":"2024-04-25T01:21:51.119159+0000","last_became_active":"2024-04-25T01:21:46.085479+0000","last_became_peered":"2024-04-25T01:21:46.085479+0000","last_unstale":"2024-04-25T01:21:51.119159+0000","last_undegraded":"2024-04-25T01:21:51.119159+0000","last_fullsized":"2024-04-25T01:21:51.119159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:02.493076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:45.085085+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:45.084796+0000","last_became_peered":"2024-04-25T01:21:45.084796+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:25.950316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118843+0000","last_change":"2024-04-25T01:21:50.123379+0000","last_active":"2024-04-25T01:21:51.118843+0000","last_peered":"2024-04-25T01:21:51.118843+0000","last_clean":"2024-04-25T01:21:51.118843+0000","last_became_active":"2024-04-25T01:21:50.123057+0000","last_became_peered":"2024-04-25T01:21:50.123057+0000","last_unstale":"2024-04-25T01:21:51.118843+0000","last_undegraded":"2024-04-25T01:21:51.118843+0000","last_fullsized":"2024-04-25T01:21:51.118843+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:13.828811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472820+0000","last_change":"2024-04-25T01:21:46.090075+0000","last_active":"2024-04-25T01:21:51.472820+0000","last_peered":"2024-04-25T01:21:51.472820+0000","last_clean":"2024-04-25T01:21:51.472820+0000","last_became_active":"2024-04-25T01:21:46.089800+0000","last_became_peered":"2024-04-25T01:21:46.089800+0000","last_unstale":"2024-04-25T01:21:51.472820+0000","last_undegraded":"2024-04-25T01:21:51.472820+0000","last_fullsized":"2024-04-25T01:21:51.472820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119254+0000","last_change":"2024-04-25T01:21:45.080102+0000","last_active":"2024-04-25T01:21:51.119254+0000","last_peered":"2024-04-25T01:21:51.119254+0000","last_clean":"2024-04-25T01:21:51.119254+0000","last_became_active":"2024-04-25T01:21:45.079786+0000","last_became_peered":"2024-04-25T01:21:45.079786+0000","last_unstale":"2024-04-25T01:21:51.119254+0000","last_undegraded":"2024-04-25T01:21:51.119254+0000","last_fullsized":"2024-04-25T01:21:51.119254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:51.235948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118619+0000","last_change":"2024-04-25T01:21:50.185727+0000","last_active":"2024-04-25T01:21:51.118619+0000","last_peered":"2024-04-25T01:21:51.118619+0000","last_clean":"2024-04-25T01:21:51.118619+0000","last_became_active":"2024-04-25T01:21:50.184395+0000","last_became_peered":"2024-04-25T01:21:50.184395+0000","last_unstale":"2024-04-25T01:21:51.118619+0000","last_undegraded":"2024-04-25T01:21:51.118619+0000","last_fullsized":"2024-04-25T01:21:51.118619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:05.257957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472627+0000","last_change":"2024-04-25T01:21:46.099799+0000","last_active":"2024-04-25T01:21:51.472627+0000","last_peered":"2024-04-25T01:21:51.472627+0000","last_clean":"2024-04-25T01:21:51.472627+0000","last_became_active":"2024-04-25T01:21:46.099506+0000","last_became_peered":"2024-04-25T01:21:46.099506+0000","last_unstale":"2024-04-25T01:21:51.472627+0000","last_undegraded":"2024-04-25T01:21:51.472627+0000","last_fullsized":"2024-04-25T01:21:51.472627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473591+0000","last_change":"2024-04-25T01:21:45.079966+0000","last_active":"2024-04-25T01:21:51.473591+0000","last_peered":"2024-04-25T01:21:51.473591+0000","last_clean":"2024-04-25T01:21:51.473591+0000","last_became_active":"2024-04-25T01:21:45.079689+0000","last_became_peered":"2024-04-25T01:21:45.079689+0000","last_unstale":"2024-04-25T01:21:51.473591+0000","last_undegraded":"2024-04-25T01:21:51.473591+0000","last_fullsized":"2024-04-25T01:21:51.473591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471654+0000","last_change":"2024-04-25T01:21:50.184865+0000","last_active":"2024-04-25T01:21:51.471654+0000","last_peered":"2024-04-25T01:21:51.471654+0000","last_clean":"2024-04-25T01:21:51.471654+0000","last_became_active":"2024-04-25T01:21:50.184466+0000","last_became_peered":"2024-04-25T01:21:50.184466+0000","last_unstale":"2024-04-25T01:21:51.471654+0000","last_undegraded":"2024-04-25T01:21:51.471654+0000","last_fullsized":"2024-04-25T01:21:51.471654+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:42.606728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473876+0000","last_change":"2024-04-25T01:21:46.100915+0000","last_active":"2024-04-25T01:21:51.473876+0000","last_peered":"2024-04-25T01:21:51.473876+0000","last_clean":"2024-04-25T01:21:51.473876+0000","last_became_active":"2024-04-25T01:21:46.100641+0000","last_became_peered":"2024-04-25T01:21:46.100641+0000","last_unstale":"2024-04-25T01:21:51.473876+0000","last_undegraded":"2024-04-25T01:21:51.473876+0000","last_fullsized":"2024-04-25T01:21:51.473876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472835+0000","last_change":"2024-04-25T01:21:45.085103+0000","last_active":"2024-04-25T01:21:51.472835+0000","last_peered":"2024-04-25T01:21:51.472835+0000","last_clean":"2024-04-25T01:21:51.472835+0000","last_became_active":"2024-04-25T01:21:45.084805+0000","last_became_peered":"2024-04-25T01:21:45.084805+0000","last_unstale":"2024-04-25T01:21:51.472835+0000","last_undegraded":"2024-04-25T01:21:51.472835+0000","last_fullsized":"2024-04-25T01:21:51.472835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:56.444547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471942+0000","last_change":"2024-04-25T01:21:50.184815+0000","last_active":"2024-04-25T01:21:51.471942+0000","last_peered":"2024-04-25T01:21:51.471942+0000","last_clean":"2024-04-25T01:21:51.471942+0000","last_became_active":"2024-04-25T01:21:50.184441+0000","last_became_peered":"2024-04-25T01:21:50.184441+0000","last_unstale":"2024-04-25T01:21:51.471942+0000","last_undegraded":"2024-04-25T01:21:51.471942+0000","last_fullsized":"2024-04-25T01:21:51.471942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473364+0000","last_change":"2024-04-25T01:21:46.096428+0000","last_active":"2024-04-25T01:21:51.473364+0000","last_peered":"2024-04-25T01:21:51.473364+0000","last_clean":"2024-04-25T01:21:51.473364+0000","last_became_active":"2024-04-25T01:21:46.096151+0000","last_became_peered":"2024-04-25T01:21:46.096151+0000","last_unstale":"2024-04-25T01:21:51.473364+0000","last_undegraded":"2024-04-25T01:21:51.473364+0000","last_fullsized":"2024-04-25T01:21:51.473364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:42.961329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473960+0000","last_change":"2024-04-25T01:21:45.080814+0000","last_active":"2024-04-25T01:21:51.473960+0000","last_peered":"2024-04-25T01:21:51.473960+0000","last_clean":"2024-04-25T01:21:51.473960+0000","last_became_active":"2024-04-25T01:21:45.080505+0000","last_became_peered":"2024-04-25T01:21:45.080505+0000","last_unstale":"2024-04-25T01:21:51.473960+0000","last_undegraded":"2024-04-25T01:21:51.473960+0000","last_fullsized":"2024-04-25T01:21:51.473960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473549+0000","last_change":"2024-04-25T01:21:50.129319+0000","last_active":"2024-04-25T01:21:51.473549+0000","last_peered":"2024-04-25T01:21:51.473549+0000","last_clean":"2024-04-25T01:21:51.473549+0000","last_became_active":"2024-04-25T01:21:50.128797+0000","last_became_peered":"2024-04-25T01:21:50.128797+0000","last_unstale":"2024-04-25T01:21:51.473549+0000","last_undegraded":"2024-04-25T01:21:51.473549+0000","last_fullsized":"2024-04-25T01:21:51.473549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:31.623179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119209+0000","last_change":"2024-04-25T01:21:46.079764+0000","last_active":"2024-04-25T01:21:51.119209+0000","last_peered":"2024-04-25T01:21:51.119209+0000","last_clean":"2024-04-25T01:21:51.119209+0000","last_became_active":"2024-04-25T01:21:46.079475+0000","last_became_peered":"2024-04-25T01:21:46.079475+0000","last_unstale":"2024-04-25T01:21:51.119209+0000","last_undegraded":"2024-04-25T01:21:51.119209+0000","last_fullsized":"2024-04-25T01:21:51.119209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:21.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473306+0000","last_change":"2024-04-25T01:21:45.075554+0000","last_active":"2024-04-25T01:21:51.473306+0000","last_peered":"2024-04-25T01:21:51.473306+0000","last_clean":"2024-04-25T01:21:51.473306+0000","last_became_active":"2024-04-25T01:21:45.075256+0000","last_became_peered":"2024-04-25T01:21:45.075256+0000","last_unstale":"2024-04-25T01:21:51.473306+0000","last_undegraded":"2024-04-25T01:21:51.473306+0000","last_fullsized":"2024-04-25T01:21:51.473306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473829+0000","last_change":"2024-04-25T01:21:50.131544+0000","last_active":"2024-04-25T01:21:51.473829+0000","last_peered":"2024-04-25T01:21:51.473829+0000","last_clean":"2024-04-25T01:21:51.473829+0000","last_became_active":"2024-04-25T01:21:50.131125+0000","last_became_peered":"2024-04-25T01:21:50.131125+0000","last_unstale":"2024-04-25T01:21:51.473829+0000","last_undegraded":"2024-04-25T01:21:51.473829+0000","last_fullsized":"2024-04-25T01:21:51.473829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:00.042952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472863+0000","last_change":"2024-04-25T01:21:46.094965+0000","last_active":"2024-04-25T01:21:51.472863+0000","last_peered":"2024-04-25T01:21:51.472863+0000","last_clean":"2024-04-25T01:21:51.472863+0000","last_became_active":"2024-04-25T01:21:46.094515+0000","last_became_peered":"2024-04-25T01:21:46.094515+0000","last_unstale":"2024-04-25T01:21:51.472863+0000","last_undegraded":"2024-04-25T01:21:51.472863+0000","last_fullsized":"2024-04-25T01:21:51.472863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119246+0000","last_change":"2024-04-25T01:21:45.075901+0000","last_active":"2024-04-25T01:21:51.119246+0000","last_peered":"2024-04-25T01:21:51.119246+0000","last_clean":"2024-04-25T01:21:51.119246+0000","last_became_active":"2024-04-25T01:21:45.075635+0000","last_became_peered":"2024-04-25T01:21:45.075635+0000","last_unstale":"2024-04-25T01:21:51.119246+0000","last_undegraded":"2024-04-25T01:21:51.119246+0000","last_fullsized":"2024-04-25T01:21:51.119246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473778+0000","last_change":"2024-04-25T01:21:50.129249+0000","last_active":"2024-04-25T01:21:51.473778+0000","last_peered":"2024-04-25T01:21:51.473778+0000","last_clean":"2024-04-25T01:21:51.473778+0000","last_became_active":"2024-04-25T01:21:50.128867+0000","last_became_peered":"2024-04-25T01:21:50.128867+0000","last_unstale":"2024-04-25T01:21:51.473778+0000","last_undegraded":"2024-04-25T01:21:51.473778+0000","last_fullsized":"2024-04-25T01:21:51.473778+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:50.627648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471590+0000","last_change":"2024-04-25T01:21:46.078894+0000","last_active":"2024-04-25T01:21:51.471590+0000","last_peered":"2024-04-25T01:21:51.471590+0000","last_clean":"2024-04-25T01:21:51.471590+0000","last_became_active":"2024-04-25T01:21:46.078598+0000","last_became_peered":"2024-04-25T01:21:46.078598+0000","last_unstale":"2024-04-25T01:21:51.471590+0000","last_undegraded":"2024-04-25T01:21:51.471590+0000","last_fullsized":"2024-04-25T01:21:51.471590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:57.449234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471807+0000","last_change":"2024-04-25T01:21:45.080835+0000","last_active":"2024-04-25T01:21:51.471807+0000","last_peered":"2024-04-25T01:21:51.471807+0000","last_clean":"2024-04-25T01:21:51.471807+0000","last_became_active":"2024-04-25T01:21:45.080544+0000","last_became_peered":"2024-04-25T01:21:45.080544+0000","last_unstale":"2024-04-25T01:21:51.471807+0000","last_undegraded":"2024-04-25T01:21:51.471807+0000","last_fullsized":"2024-04-25T01:21:51.471807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472752+0000","last_change":"2024-04-25T01:21:50.182264+0000","last_active":"2024-04-25T01:21:51.472752+0000","last_peered":"2024-04-25T01:21:51.472752+0000","last_clean":"2024-04-25T01:21:51.472752+0000","last_became_active":"2024-04-25T01:21:50.181854+0000","last_became_peered":"2024-04-25T01:21:50.181854+0000","last_unstale":"2024-04-25T01:21:51.472752+0000","last_undegraded":"2024-04-25T01:21:51.472752+0000","last_fullsized":"2024-04-25T01:21:51.472752+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:28.383661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:46.102152+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:46.101917+0000","last_became_peered":"2024-04-25T01:21:46.101917+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473111+0000","last_change":"2024-04-25T01:21:45.070971+0000","last_active":"2024-04-25T01:21:51.473111+0000","last_peered":"2024-04-25T01:21:51.473111+0000","last_clean":"2024-04-25T01:21:51.473111+0000","last_became_active":"2024-04-25T01:21:45.070626+0000","last_became_peered":"2024-04-25T01:21:45.070626+0000","last_unstale":"2024-04-25T01:21:51.473111+0000","last_undegraded":"2024-04-25T01:21:51.473111+0000","last_fullsized":"2024-04-25T01:21:51.473111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471866+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:21:51.471866+0000","last_peered":"2024-04-25T01:21:51.471866+0000","last_clean":"2024-04-25T01:21:51.471866+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:21:51.471866+0000","last_undegraded":"2024-04-25T01:21:51.471866+0000","last_fullsized":"2024-04-25T01:21:51.471866+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118918+0000","last_change":"2024-04-25T01:21:50.185591+0000","last_active":"2024-04-25T01:21:51.118918+0000","last_peered":"2024-04-25T01:21:51.118918+0000","last_clean":"2024-04-25T01:21:51.118918+0000","last_became_active":"2024-04-25T01:21:50.185206+0000","last_became_peered":"2024-04-25T01:21:50.185206+0000","last_unstale":"2024-04-25T01:21:51.118918+0000","last_undegraded":"2024-04-25T01:21:51.118918+0000","last_fullsized":"2024-04-25T01:21:51.118918+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.077404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472462+0000","last_change":"2024-04-25T01:21:46.104388+0000","last_active":"2024-04-25T01:21:51.472462+0000","last_peered":"2024-04-25T01:21:51.472462+0000","last_clean":"2024-04-25T01:21:51.472462+0000","last_became_active":"2024-04-25T01:21:46.103864+0000","last_became_peered":"2024-04-25T01:21:46.103864+0000","last_unstale":"2024-04-25T01:21:51.472462+0000","last_undegraded":"2024-04-25T01:21:51.472462+0000","last_fullsized":"2024-04-25T01:21:51.472462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:46.038059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119490+0000","last_change":"2024-04-25T01:21:45.078985+0000","last_active":"2024-04-25T01:21:51.119490+0000","last_peered":"2024-04-25T01:21:51.119490+0000","last_clean":"2024-04-25T01:21:51.119490+0000","last_became_active":"2024-04-25T01:21:45.078782+0000","last_became_peered":"2024-04-25T01:21:45.078782+0000","last_unstale":"2024-04-25T01:21:51.119490+0000","last_undegraded":"2024-04-25T01:21:51.119490+0000","last_fullsized":"2024-04-25T01:21:51.119490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:51.235948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118175+0000","last_change":"2024-04-25T01:21:50.182278+0000","last_active":"2024-04-25T01:21:51.118175+0000","last_peered":"2024-04-25T01:21:51.118175+0000","last_clean":"2024-04-25T01:21:51.118175+0000","last_became_active":"2024-04-25T01:21:50.181838+0000","last_became_peered":"2024-04-25T01:21:50.181838+0000","last_unstale":"2024-04-25T01:21:51.118175+0000","last_undegraded":"2024-04-25T01:21:51.118175+0000","last_fullsized":"2024-04-25T01:21:51.118175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:13.828811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473013+0000","last_change":"2024-04-25T01:21:46.096433+0000","last_active":"2024-04-25T01:21:51.473013+0000","last_peered":"2024-04-25T01:21:51.473013+0000","last_clean":"2024-04-25T01:21:51.473013+0000","last_became_active":"2024-04-25T01:21:46.096197+0000","last_became_peered":"2024-04-25T01:21:46.096197+0000","last_unstale":"2024-04-25T01:21:51.473013+0000","last_undegraded":"2024-04-25T01:21:51.473013+0000","last_fullsized":"2024-04-25T01:21:51.473013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:38.567383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473546+0000","last_change":"2024-04-25T01:21:45.072096+0000","last_active":"2024-04-25T01:21:51.473546+0000","last_peered":"2024-04-25T01:21:51.473546+0000","last_clean":"2024-04-25T01:21:51.473546+0000","last_became_active":"2024-04-25T01:21:45.071811+0000","last_became_peered":"2024-04-25T01:21:45.071811+0000","last_unstale":"2024-04-25T01:21:51.473546+0000","last_undegraded":"2024-04-25T01:21:51.473546+0000","last_fullsized":"2024-04-25T01:21:51.473546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472183+0000","last_change":"2024-04-25T01:21:50.123158+0000","last_active":"2024-04-25T01:21:51.472183+0000","last_peered":"2024-04-25T01:21:51.472183+0000","last_clean":"2024-04-25T01:21:51.472183+0000","last_became_active":"2024-04-25T01:21:50.122275+0000","last_became_peered":"2024-04-25T01:21:50.122275+0000","last_unstale":"2024-04-25T01:21:51.472183+0000","last_undegraded":"2024-04-25T01:21:51.472183+0000","last_fullsized":"2024-04-25T01:21:51.472183+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:49.064038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472847+0000","last_change":"2024-04-25T01:21:46.105159+0000","last_active":"2024-04-25T01:21:51.472847+0000","last_peered":"2024-04-25T01:21:51.472847+0000","last_clean":"2024-04-25T01:21:51.472847+0000","last_became_active":"2024-04-25T01:21:46.104876+0000","last_became_peered":"2024-04-25T01:21:46.104876+0000","last_unstale":"2024-04-25T01:21:51.472847+0000","last_undegraded":"2024-04-25T01:21:51.472847+0000","last_fullsized":"2024-04-25T01:21:51.472847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:38.567383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119570+0000","last_change":"2024-04-25T01:21:45.076578+0000","last_active":"2024-04-25T01:21:51.119570+0000","last_peered":"2024-04-25T01:21:51.119570+0000","last_clean":"2024-04-25T01:21:51.119570+0000","last_became_active":"2024-04-25T01:21:45.076022+0000","last_became_peered":"2024-04-25T01:21:45.076022+0000","last_unstale":"2024-04-25T01:21:51.119570+0000","last_undegraded":"2024-04-25T01:21:51.119570+0000","last_fullsized":"2024-04-25T01:21:51.119570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471952+0000","last_change":"2024-04-25T01:21:50.182219+0000","last_active":"2024-04-25T01:21:51.471952+0000","last_peered":"2024-04-25T01:21:51.471952+0000","last_clean":"2024-04-25T01:21:51.471952+0000","last_became_active":"2024-04-25T01:21:50.181820+0000","last_became_peered":"2024-04-25T01:21:50.181820+0000","last_unstale":"2024-04-25T01:21:51.471952+0000","last_undegraded":"2024-04-25T01:21:51.471952+0000","last_fullsized":"2024-04-25T01:21:51.471952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:20.747339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472586+0000","last_change":"2024-04-25T01:21:46.100017+0000","last_active":"2024-04-25T01:21:51.472586+0000","last_peered":"2024-04-25T01:21:51.472586+0000","last_clean":"2024-04-25T01:21:51.472586+0000","last_became_active":"2024-04-25T01:21:46.099745+0000","last_became_peered":"2024-04-25T01:21:46.099745+0000","last_unstale":"2024-04-25T01:21:51.472586+0000","last_undegraded":"2024-04-25T01:21:51.472586+0000","last_fullsized":"2024-04-25T01:21:51.472586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:30.138160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118932+0000","last_change":"2024-04-25T01:21:45.081659+0000","last_active":"2024-04-25T01:21:51.118932+0000","last_peered":"2024-04-25T01:21:51.118932+0000","last_clean":"2024-04-25T01:21:51.118932+0000","last_became_active":"2024-04-25T01:21:45.081320+0000","last_became_peered":"2024-04-25T01:21:45.081320+0000","last_unstale":"2024-04-25T01:21:51.118932+0000","last_undegraded":"2024-04-25T01:21:51.118932+0000","last_fullsized":"2024-04-25T01:21:51.118932+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:11:25.817031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118175+0000","last_change":"2024-04-25T01:21:50.122840+0000","last_active":"2024-04-25T01:21:51.118175+0000","last_peered":"2024-04-25T01:21:51.118175+0000","last_clean":"2024-04-25T01:21:51.118175+0000","last_became_active":"2024-04-25T01:21:50.122184+0000","last_became_peered":"2024-04-25T01:21:50.122184+0000","last_unstale":"2024-04-25T01:21:51.118175+0000","last_undegraded":"2024-04-25T01:21:51.118175+0000","last_fullsized":"2024-04-25T01:21:51.118175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:27.010679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472223+0000","last_change":"2024-04-25T01:21:46.105988+0000","last_active":"2024-04-25T01:21:51.472223+0000","last_peered":"2024-04-25T01:21:51.472223+0000","last_clean":"2024-04-25T01:21:51.472223+0000","last_became_active":"2024-04-25T01:21:46.105479+0000","last_became_peered":"2024-04-25T01:21:46.105479+0000","last_unstale":"2024-04-25T01:21:51.472223+0000","last_undegraded":"2024-04-25T01:21:51.472223+0000","last_fullsized":"2024-04-25T01:21:51.472223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:45.024693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119108+0000","last_change":"2024-04-25T01:21:45.074194+0000","last_active":"2024-04-25T01:21:51.119108+0000","last_peered":"2024-04-25T01:21:51.119108+0000","last_clean":"2024-04-25T01:21:51.119108+0000","last_became_active":"2024-04-25T01:21:45.073923+0000","last_became_peered":"2024-04-25T01:21:45.073923+0000","last_unstale":"2024-04-25T01:21:51.119108+0000","last_undegraded":"2024-04-25T01:21:51.119108+0000","last_fullsized":"2024-04-25T01:21:51.119108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472699+0000","last_change":"2024-04-25T01:21:50.126685+0000","last_active":"2024-04-25T01:21:51.472699+0000","last_peered":"2024-04-25T01:21:51.472699+0000","last_clean":"2024-04-25T01:21:51.472699+0000","last_became_active":"2024-04-25T01:21:50.126183+0000","last_became_peered":"2024-04-25T01:21:50.126183+0000","last_unstale":"2024-04-25T01:21:51.472699+0000","last_undegraded":"2024-04-25T01:21:51.472699+0000","last_fullsized":"2024-04-25T01:21:51.472699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.177506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118469+0000","last_change":"2024-04-25T01:21:46.096531+0000","last_active":"2024-04-25T01:21:51.118469+0000","last_peered":"2024-04-25T01:21:51.118469+0000","last_clean":"2024-04-25T01:21:51.118469+0000","last_became_active":"2024-04-25T01:21:46.096250+0000","last_became_peered":"2024-04-25T01:21:46.096250+0000","last_unstale":"2024-04-25T01:21:51.118469+0000","last_undegraded":"2024-04-25T01:21:51.118469+0000","last_fullsized":"2024-04-25T01:21:51.118469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:55.076328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.117985+0000","last_change":"2024-04-25T01:21:45.079574+0000","last_active":"2024-04-25T01:21:51.117985+0000","last_peered":"2024-04-25T01:21:51.117985+0000","last_clean":"2024-04-25T01:21:51.117985+0000","last_became_active":"2024-04-25T01:21:45.079306+0000","last_became_peered":"2024-04-25T01:21:45.079306+0000","last_unstale":"2024-04-25T01:21:51.117985+0000","last_undegraded":"2024-04-25T01:21:51.117985+0000","last_fullsized":"2024-04-25T01:21:51.117985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472488+0000","last_change":"2024-04-25T01:21:50.187252+0000","last_active":"2024-04-25T01:21:51.472488+0000","last_peered":"2024-04-25T01:21:51.472488+0000","last_clean":"2024-04-25T01:21:51.472488+0000","last_became_active":"2024-04-25T01:21:50.186509+0000","last_became_peered":"2024-04-25T01:21:50.186509+0000","last_unstale":"2024-04-25T01:21:51.472488+0000","last_undegraded":"2024-04-25T01:21:51.472488+0000","last_fullsized":"2024-04-25T01:21:51.472488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:15.374624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472403+0000","last_change":"2024-04-25T01:21:46.104318+0000","last_active":"2024-04-25T01:21:51.472403+0000","last_peered":"2024-04-25T01:21:51.472403+0000","last_clean":"2024-04-25T01:21:51.472403+0000","last_became_active":"2024-04-25T01:21:46.103937+0000","last_became_peered":"2024-04-25T01:21:46.103937+0000","last_unstale":"2024-04-25T01:21:51.472403+0000","last_undegraded":"2024-04-25T01:21:51.472403+0000","last_fullsized":"2024-04-25T01:21:51.472403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:27.802903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473263+0000","last_change":"2024-04-25T01:21:45.076486+0000","last_active":"2024-04-25T01:21:51.473263+0000","last_peered":"2024-04-25T01:21:51.473263+0000","last_clean":"2024-04-25T01:21:51.473263+0000","last_became_active":"2024-04-25T01:21:45.076164+0000","last_became_peered":"2024-04-25T01:21:45.076164+0000","last_unstale":"2024-04-25T01:21:51.473263+0000","last_undegraded":"2024-04-25T01:21:51.473263+0000","last_fullsized":"2024-04-25T01:21:51.473263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472905+0000","last_change":"2024-04-25T01:21:50.182316+0000","last_active":"2024-04-25T01:21:51.472905+0000","last_peered":"2024-04-25T01:21:51.472905+0000","last_clean":"2024-04-25T01:21:51.472905+0000","last_became_active":"2024-04-25T01:21:50.181918+0000","last_became_peered":"2024-04-25T01:21:50.181918+0000","last_unstale":"2024-04-25T01:21:51.472905+0000","last_undegraded":"2024-04-25T01:21:51.472905+0000","last_fullsized":"2024-04-25T01:21:51.472905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:37.041783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472461+0000","last_change":"2024-04-25T01:21:46.100833+0000","last_active":"2024-04-25T01:21:51.472461+0000","last_peered":"2024-04-25T01:21:51.472461+0000","last_clean":"2024-04-25T01:21:51.472461+0000","last_became_active":"2024-04-25T01:21:46.100591+0000","last_became_peered":"2024-04-25T01:21:46.100591+0000","last_unstale":"2024-04-25T01:21:51.472461+0000","last_undegraded":"2024-04-25T01:21:51.472461+0000","last_fullsized":"2024-04-25T01:21:51.472461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:36.537071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472415+0000","last_change":"2024-04-25T01:21:45.078469+0000","last_active":"2024-04-25T01:21:51.472415+0000","last_peered":"2024-04-25T01:21:51.472415+0000","last_clean":"2024-04-25T01:21:51.472415+0000","last_became_active":"2024-04-25T01:21:45.078231+0000","last_became_peered":"2024-04-25T01:21:45.078231+0000","last_unstale":"2024-04-25T01:21:51.472415+0000","last_undegraded":"2024-04-25T01:21:51.472415+0000","last_fullsized":"2024-04-25T01:21:51.472415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:23.371789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472786+0000","last_change":"2024-04-25T01:21:50.185207+0000","last_active":"2024-04-25T01:21:51.472786+0000","last_peered":"2024-04-25T01:21:51.472786+0000","last_clean":"2024-04-25T01:21:51.472786+0000","last_became_active":"2024-04-25T01:21:50.184793+0000","last_became_peered":"2024-04-25T01:21:50.184793+0000","last_unstale":"2024-04-25T01:21:51.472786+0000","last_undegraded":"2024-04-25T01:21:51.472786+0000","last_fullsized":"2024-04-25T01:21:51.472786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:15.374624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472438+0000","last_change":"2024-04-25T01:21:46.104460+0000","last_active":"2024-04-25T01:21:51.472438+0000","last_peered":"2024-04-25T01:21:51.472438+0000","last_clean":"2024-04-25T01:21:51.472438+0000","last_became_active":"2024-04-25T01:21:46.104163+0000","last_became_peered":"2024-04-25T01:21:46.104163+0000","last_unstale":"2024-04-25T01:21:51.472438+0000","last_undegraded":"2024-04-25T01:21:51.472438+0000","last_fullsized":"2024-04-25T01:21:51.472438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:27.583834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:55.386967+0000","last_change":"2024-04-25T01:21:45.076242+0000","last_active":"2024-04-25T01:21:55.386967+0000","last_peered":"2024-04-25T01:21:55.386967+0000","last_clean":"2024-04-25T01:21:55.386967+0000","last_became_active":"2024-04-25T01:21:45.075952+0000","last_became_peered":"2024-04-25T01:21:45.075952+0000","last_unstale":"2024-04-25T01:21:55.386967+0000","last_undegraded":"2024-04-25T01:21:55.386967+0000","last_fullsized":"2024-04-25T01:21:55.386967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:52.481389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473757+0000","last_change":"2024-04-25T01:21:50.128837+0000","last_active":"2024-04-25T01:21:51.473757+0000","last_peered":"2024-04-25T01:21:51.473757+0000","last_clean":"2024-04-25T01:21:51.473757+0000","last_became_active":"2024-04-25T01:21:50.128462+0000","last_became_peered":"2024-04-25T01:21:50.128462+0000","last_unstale":"2024-04-25T01:21:51.473757+0000","last_undegraded":"2024-04-25T01:21:51.473757+0000","last_fullsized":"2024-04-25T01:21:51.473757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:42.606728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118967+0000","last_change":"2024-04-25T01:21:46.100341+0000","last_active":"2024-04-25T01:21:51.118967+0000","last_peered":"2024-04-25T01:21:51.118967+0000","last_clean":"2024-04-25T01:21:51.118967+0000","last_became_active":"2024-04-25T01:21:46.099961+0000","last_became_peered":"2024-04-25T01:21:46.099961+0000","last_unstale":"2024-04-25T01:21:51.118967+0000","last_undegraded":"2024-04-25T01:21:51.118967+0000","last_fullsized":"2024-04-25T01:21:51.118967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:15.176543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473450+0000","last_change":"2024-04-25T01:21:45.077748+0000","last_active":"2024-04-25T01:21:51.473450+0000","last_peered":"2024-04-25T01:21:51.473450+0000","last_clean":"2024-04-25T01:21:51.473450+0000","last_became_active":"2024-04-25T01:21:45.077494+0000","last_became_peered":"2024-04-25T01:21:45.077494+0000","last_unstale":"2024-04-25T01:21:51.473450+0000","last_undegraded":"2024-04-25T01:21:51.473450+0000","last_fullsized":"2024-04-25T01:21:51.473450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:25.950316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118695+0000","last_change":"2024-04-25T01:21:50.126572+0000","last_active":"2024-04-25T01:21:51.118695+0000","last_peered":"2024-04-25T01:21:51.118695+0000","last_clean":"2024-04-25T01:21:51.118695+0000","last_became_active":"2024-04-25T01:21:50.126013+0000","last_became_peered":"2024-04-25T01:21:50.126013+0000","last_unstale":"2024-04-25T01:21:51.118695+0000","last_undegraded":"2024-04-25T01:21:51.118695+0000","last_fullsized":"2024-04-25T01:21:51.118695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:19.877429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471878+0000","last_change":"2024-04-25T01:21:46.091135+0000","last_active":"2024-04-25T01:21:51.471878+0000","last_peered":"2024-04-25T01:21:51.471878+0000","last_clean":"2024-04-25T01:21:51.471878+0000","last_became_active":"2024-04-25T01:21:46.090493+0000","last_became_peered":"2024-04-25T01:21:46.090493+0000","last_unstale":"2024-04-25T01:21:51.471878+0000","last_undegraded":"2024-04-25T01:21:51.471878+0000","last_fullsized":"2024-04-25T01:21:51.471878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472529+0000","last_change":"2024-04-25T01:21:45.075179+0000","last_active":"2024-04-25T01:21:51.472529+0000","last_peered":"2024-04-25T01:21:51.472529+0000","last_clean":"2024-04-25T01:21:51.472529+0000","last_became_active":"2024-04-25T01:21:45.074903+0000","last_became_peered":"2024-04-25T01:21:45.074903+0000","last_unstale":"2024-04-25T01:21:51.472529+0000","last_undegraded":"2024-04-25T01:21:51.472529+0000","last_fullsized":"2024-04-25T01:21:51.472529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471819+0000","last_change":"2024-04-25T01:21:50.126406+0000","last_active":"2024-04-25T01:21:51.471819+0000","last_peered":"2024-04-25T01:21:51.471819+0000","last_clean":"2024-04-25T01:21:51.471819+0000","last_became_active":"2024-04-25T01:21:50.126029+0000","last_became_peered":"2024-04-25T01:21:50.126029+0000","last_unstale":"2024-04-25T01:21:51.471819+0000","last_undegraded":"2024-04-25T01:21:51.471819+0000","last_fullsized":"2024-04-25T01:21:51.471819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:38.806941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472398+0000","last_change":"2024-04-25T01:21:46.100746+0000","last_active":"2024-04-25T01:21:51.472398+0000","last_peered":"2024-04-25T01:21:51.472398+0000","last_clean":"2024-04-25T01:21:51.472398+0000","last_became_active":"2024-04-25T01:21:46.100475+0000","last_became_peered":"2024-04-25T01:21:46.100475+0000","last_unstale":"2024-04-25T01:21:51.472398+0000","last_undegraded":"2024-04-25T01:21:51.472398+0000","last_fullsized":"2024-04-25T01:21:51.472398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:27.802903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119162+0000","last_change":"2024-04-25T01:21:45.082455+0000","last_active":"2024-04-25T01:21:51.119162+0000","last_peered":"2024-04-25T01:21:51.119162+0000","last_clean":"2024-04-25T01:21:51.119162+0000","last_became_active":"2024-04-25T01:21:45.082135+0000","last_became_peered":"2024-04-25T01:21:45.082135+0000","last_unstale":"2024-04-25T01:21:51.119162+0000","last_undegraded":"2024-04-25T01:21:51.119162+0000","last_fullsized":"2024-04-25T01:21:51.119162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:06.296153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471896+0000","last_change":"2024-04-25T01:21:50.184859+0000","last_active":"2024-04-25T01:21:51.471896+0000","last_peered":"2024-04-25T01:21:51.471896+0000","last_clean":"2024-04-25T01:21:51.471896+0000","last_became_active":"2024-04-25T01:21:50.184475+0000","last_became_peered":"2024-04-25T01:21:50.184475+0000","last_unstale":"2024-04-25T01:21:51.471896+0000","last_undegraded":"2024-04-25T01:21:51.471896+0000","last_fullsized":"2024-04-25T01:21:51.471896+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:00.042952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472589+0000","last_change":"2024-04-25T01:21:50.184894+0000","last_active":"2024-04-25T01:21:51.472589+0000","last_peered":"2024-04-25T01:21:51.472589+0000","last_clean":"2024-04-25T01:21:51.472589+0000","last_became_active":"2024-04-25T01:21:50.184392+0000","last_became_peered":"2024-04-25T01:21:50.184392+0000","last_unstale":"2024-04-25T01:21:51.472589+0000","last_undegraded":"2024-04-25T01:21:51.472589+0000","last_fullsized":"2024-04-25T01:21:51.472589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:31.842249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472948+0000","last_change":"2024-04-25T01:21:46.092888+0000","last_active":"2024-04-25T01:21:51.472948+0000","last_peered":"2024-04-25T01:21:51.472948+0000","last_clean":"2024-04-25T01:21:51.472948+0000","last_became_active":"2024-04-25T01:21:46.092607+0000","last_became_peered":"2024-04-25T01:21:46.092607+0000","last_unstale":"2024-04-25T01:21:51.472948+0000","last_undegraded":"2024-04-25T01:21:51.472948+0000","last_fullsized":"2024-04-25T01:21:51.472948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:22.971333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472329+0000","last_change":"2024-04-25T01:21:45.070532+0000","last_active":"2024-04-25T01:21:51.472329+0000","last_peered":"2024-04-25T01:21:51.472329+0000","last_clean":"2024-04-25T01:21:51.472329+0000","last_became_active":"2024-04-25T01:21:45.070256+0000","last_became_peered":"2024-04-25T01:21:45.070256+0000","last_unstale":"2024-04-25T01:21:51.472329+0000","last_undegraded":"2024-04-25T01:21:51.472329+0000","last_fullsized":"2024-04-25T01:21:51.472329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119151+0000","last_change":"2024-04-25T01:21:50.130056+0000","last_active":"2024-04-25T01:21:51.119151+0000","last_peered":"2024-04-25T01:21:51.119151+0000","last_clean":"2024-04-25T01:21:51.119151+0000","last_became_active":"2024-04-25T01:21:50.129541+0000","last_became_peered":"2024-04-25T01:21:50.129541+0000","last_unstale":"2024-04-25T01:21:51.119151+0000","last_undegraded":"2024-04-25T01:21:51.119151+0000","last_fullsized":"2024-04-25T01:21:51.119151+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471871+0000","last_change":"2024-04-25T01:21:46.083728+0000","last_active":"2024-04-25T01:21:51.471871+0000","last_peered":"2024-04-25T01:21:51.471871+0000","last_clean":"2024-04-25T01:21:51.471871+0000","last_became_active":"2024-04-25T01:21:46.083458+0000","last_became_peered":"2024-04-25T01:21:46.083458+0000","last_unstale":"2024-04-25T01:21:51.471871+0000","last_undegraded":"2024-04-25T01:21:51.471871+0000","last_fullsized":"2024-04-25T01:21:51.471871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472093+0000","last_change":"2024-04-25T01:21:45.077308+0000","last_active":"2024-04-25T01:21:51.472093+0000","last_peered":"2024-04-25T01:21:51.472093+0000","last_clean":"2024-04-25T01:21:51.472093+0000","last_became_active":"2024-04-25T01:21:45.076815+0000","last_became_peered":"2024-04-25T01:21:45.076815+0000","last_unstale":"2024-04-25T01:21:51.472093+0000","last_undegraded":"2024-04-25T01:21:51.472093+0000","last_fullsized":"2024-04-25T01:21:51.472093+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:43.833096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118432+0000","last_change":"2024-04-25T01:21:45.079268+0000","last_active":"2024-04-25T01:21:51.118432+0000","last_peered":"2024-04-25T01:21:51.118432+0000","last_clean":"2024-04-25T01:21:51.118432+0000","last_became_active":"2024-04-25T01:21:45.078934+0000","last_became_peered":"2024-04-25T01:21:45.078934+0000","last_unstale":"2024-04-25T01:21:51.118432+0000","last_undegraded":"2024-04-25T01:21:51.118432+0000","last_fullsized":"2024-04-25T01:21:51.118432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472316+0000","last_change":"2024-04-25T01:21:46.086329+0000","last_active":"2024-04-25T01:21:51.472316+0000","last_peered":"2024-04-25T01:21:51.472316+0000","last_clean":"2024-04-25T01:21:51.472316+0000","last_became_active":"2024-04-25T01:21:46.086069+0000","last_became_peered":"2024-04-25T01:21:46.086069+0000","last_unstale":"2024-04-25T01:21:51.472316+0000","last_undegraded":"2024-04-25T01:21:51.472316+0000","last_fullsized":"2024-04-25T01:21:51.472316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119441+0000","last_change":"2024-04-25T01:21:50.122334+0000","last_active":"2024-04-25T01:21:51.119441+0000","last_peered":"2024-04-25T01:21:51.119441+0000","last_clean":"2024-04-25T01:21:51.119441+0000","last_became_active":"2024-04-25T01:21:50.121974+0000","last_became_peered":"2024-04-25T01:21:50.121974+0000","last_unstale":"2024-04-25T01:21:51.119441+0000","last_undegraded":"2024-04-25T01:21:51.119441+0000","last_fullsized":"2024-04-25T01:21:51.119441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:59.115674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473488+0000","last_change":"2024-04-25T01:21:45.082740+0000","last_active":"2024-04-25T01:21:51.473488+0000","last_peered":"2024-04-25T01:21:51.473488+0000","last_clean":"2024-04-25T01:21:51.473488+0000","last_became_active":"2024-04-25T01:21:45.082439+0000","last_became_peered":"2024-04-25T01:21:45.082439+0000","last_unstale":"2024-04-25T01:21:51.473488+0000","last_undegraded":"2024-04-25T01:21:51.473488+0000","last_fullsized":"2024-04-25T01:21:51.473488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:52.481389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118237+0000","last_change":"2024-04-25T01:21:46.088483+0000","last_active":"2024-04-25T01:21:51.118237+0000","last_peered":"2024-04-25T01:21:51.118237+0000","last_clean":"2024-04-25T01:21:51.118237+0000","last_became_active":"2024-04-25T01:21:46.088113+0000","last_became_peered":"2024-04-25T01:21:46.088113+0000","last_unstale":"2024-04-25T01:21:51.118237+0000","last_undegraded":"2024-04-25T01:21:51.118237+0000","last_fullsized":"2024-04-25T01:21:51.118237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:22.055684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472316+0000","last_change":"2024-04-25T01:21:50.125319+0000","last_active":"2024-04-25T01:21:51.472316+0000","last_peered":"2024-04-25T01:21:51.472316+0000","last_clean":"2024-04-25T01:21:51.472316+0000","last_became_active":"2024-04-25T01:21:50.124926+0000","last_became_peered":"2024-04-25T01:21:50.124926+0000","last_unstale":"2024-04-25T01:21:51.472316+0000","last_undegraded":"2024-04-25T01:21:51.472316+0000","last_fullsized":"2024-04-25T01:21:51.472316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:20.747339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118296+0000","last_change":"2024-04-25T01:21:46.093690+0000","last_active":"2024-04-25T01:21:51.118296+0000","last_peered":"2024-04-25T01:21:51.118296+0000","last_clean":"2024-04-25T01:21:51.118296+0000","last_became_active":"2024-04-25T01:21:46.093392+0000","last_became_peered":"2024-04-25T01:21:46.093392+0000","last_unstale":"2024-04-25T01:21:51.118296+0000","last_undegraded":"2024-04-25T01:21:51.118296+0000","last_fullsized":"2024-04-25T01:21:51.118296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119144+0000","last_change":"2024-04-25T01:21:46.089771+0000","last_active":"2024-04-25T01:21:51.119144+0000","last_peered":"2024-04-25T01:21:51.119144+0000","last_clean":"2024-04-25T01:21:51.119144+0000","last_became_active":"2024-04-25T01:21:46.089489+0000","last_became_peered":"2024-04-25T01:21:46.089489+0000","last_unstale":"2024-04-25T01:21:51.119144+0000","last_undegraded":"2024-04-25T01:21:51.119144+0000","last_fullsized":"2024-04-25T01:21:51.119144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:16.213667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473158+0000","last_change":"2024-04-25T01:21:46.099812+0000","last_active":"2024-04-25T01:21:51.473158+0000","last_peered":"2024-04-25T01:21:51.473158+0000","last_clean":"2024-04-25T01:21:51.473158+0000","last_became_active":"2024-04-25T01:21:46.099508+0000","last_became_peered":"2024-04-25T01:21:46.099508+0000","last_unstale":"2024-04-25T01:21:51.473158+0000","last_undegraded":"2024-04-25T01:21:51.473158+0000","last_fullsized":"2024-04-25T01:21:51.473158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:45.024693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472233+0000","last_change":"2024-04-25T01:21:46.083637+0000","last_active":"2024-04-25T01:21:51.472233+0000","last_peered":"2024-04-25T01:21:51.472233+0000","last_clean":"2024-04-25T01:21:51.472233+0000","last_became_active":"2024-04-25T01:21:46.083371+0000","last_became_peered":"2024-04-25T01:21:46.083371+0000","last_unstale":"2024-04-25T01:21:51.472233+0000","last_undegraded":"2024-04-25T01:21:51.472233+0000","last_fullsized":"2024-04-25T01:21:51.472233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473043+0000","last_change":"2024-04-25T01:21:46.087822+0000","last_active":"2024-04-25T01:21:51.473043+0000","last_peered":"2024-04-25T01:21:51.473043+0000","last_clean":"2024-04-25T01:21:51.473043+0000","last_became_active":"2024-04-25T01:21:46.087555+0000","last_became_peered":"2024-04-25T01:21:46.087555+0000","last_unstale":"2024-04-25T01:21:51.473043+0000","last_undegraded":"2024-04-25T01:21:51.473043+0000","last_fullsized":"2024-04-25T01:21:51.473043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473261+0000","last_change":"2024-04-25T01:21:46.092978+0000","last_active":"2024-04-25T01:21:51.473261+0000","last_peered":"2024-04-25T01:21:51.473261+0000","last_clean":"2024-04-25T01:21:51.473261+0000","last_became_active":"2024-04-25T01:21:46.092720+0000","last_became_peered":"2024-04-25T01:21:46.092720+0000","last_unstale":"2024-04-25T01:21:51.473261+0000","last_undegraded":"2024-04-25T01:21:51.473261+0000","last_fullsized":"2024-04-25T01:21:51.473261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119314+0000","last_change":"2024-04-25T01:21:46.100969+0000","last_active":"2024-04-25T01:21:51.119314+0000","last_peered":"2024-04-25T01:21:51.119314+0000","last_clean":"2024-04-25T01:21:51.119314+0000","last_became_active":"2024-04-25T01:21:46.100687+0000","last_became_peered":"2024-04-25T01:21:46.100687+0000","last_unstale":"2024-04-25T01:21:51.119314+0000","last_undegraded":"2024-04-25T01:21:51.119314+0000","last_fullsized":"2024-04-25T01:21:51.119314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472539+0000","last_change":"2024-04-25T01:21:46.087688+0000","last_active":"2024-04-25T01:21:51.472539+0000","last_peered":"2024-04-25T01:21:51.472539+0000","last_clean":"2024-04-25T01:21:51.472539+0000","last_became_active":"2024-04-25T01:21:46.087422+0000","last_became_peered":"2024-04-25T01:21:46.087422+0000","last_unstale":"2024-04-25T01:21:51.472539+0000","last_undegraded":"2024-04-25T01:21:51.472539+0000","last_fullsized":"2024-04-25T01:21:51.472539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472217+0000","last_change":"2024-04-25T01:21:46.096585+0000","last_active":"2024-04-25T01:21:51.472217+0000","last_peered":"2024-04-25T01:21:51.472217+0000","last_clean":"2024-04-25T01:21:51.472217+0000","last_became_active":"2024-04-25T01:21:46.096299+0000","last_became_peered":"2024-04-25T01:21:46.096299+0000","last_unstale":"2024-04-25T01:21:51.472217+0000","last_undegraded":"2024-04-25T01:21:51.472217+0000","last_fullsized":"2024-04-25T01:21:51.472217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472814+0000","last_change":"2024-04-25T01:21:46.085048+0000","last_active":"2024-04-25T01:21:51.472814+0000","last_peered":"2024-04-25T01:21:51.472814+0000","last_clean":"2024-04-25T01:21:51.472814+0000","last_became_active":"2024-04-25T01:21:46.084766+0000","last_became_peered":"2024-04-25T01:21:46.084766+0000","last_unstale":"2024-04-25T01:21:51.472814+0000","last_undegraded":"2024-04-25T01:21:51.472814+0000","last_fullsized":"2024-04-25T01:21:51.472814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473111+0000","last_change":"2024-04-25T01:21:46.105035+0000","last_active":"2024-04-25T01:21:51.473111+0000","last_peered":"2024-04-25T01:21:51.473111+0000","last_clean":"2024-04-25T01:21:51.473111+0000","last_became_active":"2024-04-25T01:21:46.104776+0000","last_became_peered":"2024-04-25T01:21:46.104776+0000","last_unstale":"2024-04-25T01:21:51.473111+0000","last_undegraded":"2024-04-25T01:21:51.473111+0000","last_fullsized":"2024-04-25T01:21:51.473111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471992+0000","last_change":"2024-04-25T01:21:46.100414+0000","last_active":"2024-04-25T01:21:51.471992+0000","last_peered":"2024-04-25T01:21:51.471992+0000","last_clean":"2024-04-25T01:21:51.471992+0000","last_became_active":"2024-04-25T01:21:46.100140+0000","last_became_peered":"2024-04-25T01:21:46.100140+0000","last_unstale":"2024-04-25T01:21:51.471992+0000","last_undegraded":"2024-04-25T01:21:51.471992+0000","last_fullsized":"2024-04-25T01:21:51.471992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:55.076328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118784+0000","last_change":"2024-04-25T01:21:46.092761+0000","last_active":"2024-04-25T01:21:51.118784+0000","last_peered":"2024-04-25T01:21:51.118784+0000","last_clean":"2024-04-25T01:21:51.118784+0000","last_became_active":"2024-04-25T01:21:46.092230+0000","last_became_peered":"2024-04-25T01:21:46.092230+0000","last_unstale":"2024-04-25T01:21:51.118784+0000","last_undegraded":"2024-04-25T01:21:51.118784+0000","last_fullsized":"2024-04-25T01:21:51.118784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472724+0000","last_change":"2024-04-25T01:21:46.104951+0000","last_active":"2024-04-25T01:21:51.472724+0000","last_peered":"2024-04-25T01:21:51.472724+0000","last_clean":"2024-04-25T01:21:51.472724+0000","last_became_active":"2024-04-25T01:21:46.104662+0000","last_became_peered":"2024-04-25T01:21:46.104662+0000","last_unstale":"2024-04-25T01:21:51.472724+0000","last_undegraded":"2024-04-25T01:21:51.472724+0000","last_fullsized":"2024-04-25T01:21:51.472724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:46.588303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119527+0000","last_change":"2024-04-25T01:21:46.091860+0000","last_active":"2024-04-25T01:21:51.119527+0000","last_peered":"2024-04-25T01:21:51.119527+0000","last_clean":"2024-04-25T01:21:51.119527+0000","last_became_active":"2024-04-25T01:21:46.091569+0000","last_became_peered":"2024-04-25T01:21:46.091569+0000","last_unstale":"2024-04-25T01:21:51.119527+0000","last_undegraded":"2024-04-25T01:21:51.119527+0000","last_fullsized":"2024-04-25T01:21:51.119527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:57.449234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473437+0000","last_change":"2024-04-25T01:21:46.099956+0000","last_active":"2024-04-25T01:21:51.473437+0000","last_peered":"2024-04-25T01:21:51.473437+0000","last_clean":"2024-04-25T01:21:51.473437+0000","last_became_active":"2024-04-25T01:21:46.099687+0000","last_became_peered":"2024-04-25T01:21:46.099687+0000","last_unstale":"2024-04-25T01:21:51.473437+0000","last_undegraded":"2024-04-25T01:21:51.473437+0000","last_fullsized":"2024-04-25T01:21:51.473437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:30.138160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472108+0000","last_change":"2024-04-25T01:21:46.102084+0000","last_active":"2024-04-25T01:21:51.472108+0000","last_peered":"2024-04-25T01:21:51.472108+0000","last_clean":"2024-04-25T01:21:51.472108+0000","last_became_active":"2024-04-25T01:21:46.101539+0000","last_became_peered":"2024-04-25T01:21:46.101539+0000","last_unstale":"2024-04-25T01:21:51.472108+0000","last_undegraded":"2024-04-25T01:21:51.472108+0000","last_fullsized":"2024-04-25T01:21:51.472108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473063+0000","last_change":"2024-04-25T01:21:46.089149+0000","last_active":"2024-04-25T01:21:51.473063+0000","last_peered":"2024-04-25T01:21:51.473063+0000","last_clean":"2024-04-25T01:21:51.473063+0000","last_became_active":"2024-04-25T01:21:46.088845+0000","last_became_peered":"2024-04-25T01:21:46.088845+0000","last_unstale":"2024-04-25T01:21:51.473063+0000","last_undegraded":"2024-04-25T01:21:51.473063+0000","last_fullsized":"2024-04-25T01:21:51.473063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472391+0000","last_change":"2024-04-25T01:21:46.087764+0000","last_active":"2024-04-25T01:21:51.472391+0000","last_peered":"2024-04-25T01:21:51.472391+0000","last_clean":"2024-04-25T01:21:51.472391+0000","last_became_active":"2024-04-25T01:21:46.087481+0000","last_became_peered":"2024-04-25T01:21:46.087481+0000","last_unstale":"2024-04-25T01:21:51.472391+0000","last_undegraded":"2024-04-25T01:21:51.472391+0000","last_fullsized":"2024-04-25T01:21:51.472391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:16.213667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473322+0000","last_change":"2024-04-25T01:21:46.093981+0000","last_active":"2024-04-25T01:21:51.473322+0000","last_peered":"2024-04-25T01:21:51.473322+0000","last_clean":"2024-04-25T01:21:51.473322+0000","last_became_active":"2024-04-25T01:21:46.093729+0000","last_became_peered":"2024-04-25T01:21:46.093729+0000","last_unstale":"2024-04-25T01:21:51.473322+0000","last_undegraded":"2024-04-25T01:21:51.473322+0000","last_fullsized":"2024-04-25T01:21:51.473322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118866+0000","last_change":"2024-04-25T01:21:46.094258+0000","last_active":"2024-04-25T01:21:51.118866+0000","last_peered":"2024-04-25T01:21:51.118866+0000","last_clean":"2024-04-25T01:21:51.118866+0000","last_became_active":"2024-04-25T01:21:46.093633+0000","last_became_peered":"2024-04-25T01:21:46.093633+0000","last_unstale":"2024-04-25T01:21:51.118866+0000","last_undegraded":"2024-04-25T01:21:51.118866+0000","last_fullsized":"2024-04-25T01:21:51.118866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:52.240635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118920+0000","last_change":"2024-04-25T01:21:46.093764+0000","last_active":"2024-04-25T01:21:51.118920+0000","last_peered":"2024-04-25T01:21:51.118920+0000","last_clean":"2024-04-25T01:21:51.118920+0000","last_became_active":"2024-04-25T01:21:46.093468+0000","last_became_peered":"2024-04-25T01:21:46.093468+0000","last_unstale":"2024-04-25T01:21:51.118920+0000","last_undegraded":"2024-04-25T01:21:51.118920+0000","last_fullsized":"2024-04-25T01:21:51.118920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119290+0000","last_change":"2024-04-25T01:21:46.096039+0000","last_active":"2024-04-25T01:21:51.119290+0000","last_peered":"2024-04-25T01:21:51.119290+0000","last_clean":"2024-04-25T01:21:51.119290+0000","last_became_active":"2024-04-25T01:21:46.095773+0000","last_became_peered":"2024-04-25T01:21:46.095773+0000","last_unstale":"2024-04-25T01:21:51.119290+0000","last_undegraded":"2024-04-25T01:21:51.119290+0000","last_fullsized":"2024-04-25T01:21:51.119290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472805+0000","last_change":"2024-04-25T01:21:46.106320+0000","last_active":"2024-04-25T01:21:51.472805+0000","last_peered":"2024-04-25T01:21:51.472805+0000","last_clean":"2024-04-25T01:21:51.472805+0000","last_became_active":"2024-04-25T01:21:46.106064+0000","last_became_peered":"2024-04-25T01:21:46.106064+0000","last_unstale":"2024-04-25T01:21:51.472805+0000","last_undegraded":"2024-04-25T01:21:51.472805+0000","last_fullsized":"2024-04-25T01:21:51.472805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:24.344315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471632+0000","last_change":"2024-04-25T01:21:46.093521+0000","last_active":"2024-04-25T01:21:51.471632+0000","last_peered":"2024-04-25T01:21:51.471632+0000","last_clean":"2024-04-25T01:21:51.471632+0000","last_became_active":"2024-04-25T01:21:46.093269+0000","last_became_peered":"2024-04-25T01:21:46.093269+0000","last_unstale":"2024-04-25T01:21:51.471632+0000","last_undegraded":"2024-04-25T01:21:51.471632+0000","last_fullsized":"2024-04-25T01:21:51.471632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472027+0000","last_change":"2024-04-25T01:21:46.102728+0000","last_active":"2024-04-25T01:21:51.472027+0000","last_peered":"2024-04-25T01:21:51.472027+0000","last_clean":"2024-04-25T01:21:51.472027+0000","last_became_active":"2024-04-25T01:21:46.102475+0000","last_became_peered":"2024-04-25T01:21:46.102475+0000","last_unstale":"2024-04-25T01:21:51.472027+0000","last_undegraded":"2024-04-25T01:21:51.472027+0000","last_fullsized":"2024-04-25T01:21:51.472027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472990+0000","last_change":"2024-04-25T01:21:46.101690+0000","last_active":"2024-04-25T01:21:51.472990+0000","last_peered":"2024-04-25T01:21:51.472990+0000","last_clean":"2024-04-25T01:21:51.472990+0000","last_became_active":"2024-04-25T01:21:46.101404+0000","last_became_peered":"2024-04-25T01:21:46.101404+0000","last_unstale":"2024-04-25T01:21:51.472990+0000","last_undegraded":"2024-04-25T01:21:51.472990+0000","last_fullsized":"2024-04-25T01:21:51.472990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472132+0000","last_change":"2024-04-25T01:21:46.103900+0000","last_active":"2024-04-25T01:21:51.472132+0000","last_peered":"2024-04-25T01:21:51.472132+0000","last_clean":"2024-04-25T01:21:51.472132+0000","last_became_active":"2024-04-25T01:21:46.103633+0000","last_became_peered":"2024-04-25T01:21:46.103633+0000","last_unstale":"2024-04-25T01:21:51.472132+0000","last_undegraded":"2024-04-25T01:21:51.472132+0000","last_fullsized":"2024-04-25T01:21:51.472132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:42.961329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472786+0000","last_change":"2024-04-25T01:21:46.102312+0000","last_active":"2024-04-25T01:21:51.472786+0000","last_peered":"2024-04-25T01:21:51.472786+0000","last_clean":"2024-04-25T01:21:51.472786+0000","last_became_active":"2024-04-25T01:21:46.101113+0000","last_became_peered":"2024-04-25T01:21:46.101113+0000","last_unstale":"2024-04-25T01:21:51.472786+0000","last_undegraded":"2024-04-25T01:21:51.472786+0000","last_fullsized":"2024-04-25T01:21:51.472786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:46.038059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473192+0000","last_change":"2024-04-25T01:21:46.098593+0000","last_active":"2024-04-25T01:21:51.473192+0000","last_peered":"2024-04-25T01:21:51.473192+0000","last_clean":"2024-04-25T01:21:51.473192+0000","last_became_active":"2024-04-25T01:21:46.098289+0000","last_became_peered":"2024-04-25T01:21:46.098289+0000","last_unstale":"2024-04-25T01:21:51.473192+0000","last_undegraded":"2024-04-25T01:21:51.473192+0000","last_fullsized":"2024-04-25T01:21:51.473192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473504+0000","last_change":"2024-04-25T01:21:46.093954+0000","last_active":"2024-04-25T01:21:51.473504+0000","last_peered":"2024-04-25T01:21:51.473504+0000","last_clean":"2024-04-25T01:21:51.473504+0000","last_became_active":"2024-04-25T01:21:46.093658+0000","last_became_peered":"2024-04-25T01:21:46.093658+0000","last_unstale":"2024-04-25T01:21:51.473504+0000","last_undegraded":"2024-04-25T01:21:51.473504+0000","last_fullsized":"2024-04-25T01:21:51.473504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472718+0000","last_change":"2024-04-25T01:21:46.108321+0000","last_active":"2024-04-25T01:21:51.472718+0000","last_peered":"2024-04-25T01:21:51.472718+0000","last_clean":"2024-04-25T01:21:51.472718+0000","last_became_active":"2024-04-25T01:21:46.108056+0000","last_became_peered":"2024-04-25T01:21:46.108056+0000","last_unstale":"2024-04-25T01:21:51.472718+0000","last_undegraded":"2024-04-25T01:21:51.472718+0000","last_fullsized":"2024-04-25T01:21:51.472718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:22:12.454 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:22:12.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.606+0000 7fded069e640 1 Processor -- start 2024-04-25T01:22:12.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.606+0000 7fded069e640 1 -- start start 2024-04-25T01:22:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fded069e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec8145590 0x7fdec80b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fded069e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdec80b4240 con 0x7fdec806e280 2024-04-25T01:22:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fded069e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdec80b4f60 con 0x7fdec815f9b0 2024-04-25T01:22:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fded069e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdec8160830 con 0x7fdec8145590 2024-04-25T01:22:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fdecec3b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdec806e280 0x7fdec80ce820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56564/0 (socket says 172.21.15.77:56564) 2024-04-25T01:22:12.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fdecec3b640 1 -- 172.21.15.77:0/1997851916 learned_addr learned my addr 172.21.15.77:0/1997851916 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:12.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fdece43a640 1 --2- 172.21.15.77:0/1997851916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec8145590 0x7fdec80b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fdecd438640 1 -- 172.21.15.77:0/1997851916 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4043284697 0 0) 0x7fdec80b4240 con 0x7fdec806e280 2024-04-25T01:22:12.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fdecd438640 1 -- 172.21.15.77:0/1997851916 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdeb4003520 con 0x7fdec806e280 2024-04-25T01:22:12.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fdecd438640 1 -- 172.21.15.77:0/1997851916 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2463970188 0 0) 0x7fdec80b4f60 con 0x7fdec815f9b0 2024-04-25T01:22:12.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fdece43a640 1 -- 172.21.15.77:0/1997851916 >> v1:172.21.15.121:6789/0 conn(0x7fdec815f9b0 legacy=0x7fdec80ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fdece43a640 1 -- 172.21.15.77:0/1997851916 >> v1:172.21.15.77:6789/0 conn(0x7fdec806e280 legacy=0x7fdec80ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.607+0000 7fdece43a640 1 -- 172.21.15.77:0/1997851916 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec80b39a0 con 0x7fdec8145590 2024-04-25T01:22:12.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.608+0000 7fdece43a640 1 --2- 172.21.15.77:0/1997851916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec8145590 0x7fdec80b4a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdec40097d0 tx=0x7fdec402a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8427592f560527af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:12.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.608+0000 7fdecd438640 1 -- 172.21.15.77:0/1997851916 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4037070 con 0x7fdec8145590 2024-04-25T01:22:12.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.608+0000 7fdecd438640 1 -- 172.21.15.77:0/1997851916 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec402ae30 con 0x7fdec8145590 2024-04-25T01:22:12.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.608+0000 7fdecd438640 1 -- 172.21.15.77:0/1997851916 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4032640 con 0x7fdec8145590 2024-04-25T01:22:12.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.609+0000 7fded069e640 1 -- 172.21.15.77:0/1997851916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec8145590 msgr2=0x7fdec80b4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.609+0000 7fded069e640 1 --2- 172.21.15.77:0/1997851916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec8145590 0x7fdec80b4a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdec40097d0 tx=0x7fdec402a1a0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.609+0000 7fded069e640 1 -- 172.21.15.77:0/1997851916 shutdown_connections 2024-04-25T01:22:12.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.609+0000 7fded069e640 1 --2- 172.21.15.77:0/1997851916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec8145590 0x7fdec80b4a20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.609+0000 7fded069e640 1 -- 172.21.15.77:0/1997851916 >> 172.21.15.77:0/1997851916 conn(0x7fdec80b1820 msgr2=0x7fdec80b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.609+0000 7fded069e640 1 -- 172.21.15.77:0/1997851916 shutdown_connections 2024-04-25T01:22:12.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.609+0000 7fded069e640 1 -- 172.21.15.77:0/1997851916 wait complete. 2024-04-25T01:22:12.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.610+0000 7fded069e640 1 Processor -- start 2024-04-25T01:22:12.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.610+0000 7fded069e640 1 -- start start 2024-04-25T01:22:12.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fded069e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec806e280 0x7fdec815a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fded069e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdec8145590 0x7fdec8156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fded069e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec815f9b0 0x7fdec815d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fded069e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdec80b3f60 con 0x7fdec8145590 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fded069e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdec806cde0 con 0x7fdec806e280 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fded069e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdec8145970 con 0x7fdec815f9b0 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdecdc39640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdec8145590 0x7fdec8156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdecdc39640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdec8145590 0x7fdec8156830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56576/0 (socket says 172.21.15.77:56576) 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdecdc39640 1 -- 172.21.15.77:0/3001982867 learned_addr learned my addr 172.21.15.77:0/3001982867 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdecec3b640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec815f9b0 0x7fdec815d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdece43a640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec806e280 0x7fdec815a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdecdc39640 1 -- 172.21.15.77:0/3001982867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec815f9b0 msgr2=0x7fdec815d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdecdc39640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec815f9b0 0x7fdec815d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdecdc39640 1 -- 172.21.15.77:0/3001982867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec806e280 msgr2=0x7fdec815a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdecdc39640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec806e280 0x7fdec815a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdecdc39640 1 -- 172.21.15.77:0/3001982867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec815d7a0 con 0x7fdec8145590 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdecdc39640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdec8145590 0x7fdec8156830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdeb800c410 tx=0x7fdeb800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:12.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.611+0000 7fdebeffd640 1 -- 172.21.15.77:0/3001982867 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8017070 con 0x7fdec8145590 2024-04-25T01:22:12.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.612+0000 7fdebeffd640 1 -- 172.21.15.77:0/3001982867 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb80035f0 con 0x7fdec8145590 2024-04-25T01:22:12.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.612+0000 7fded069e640 1 -- 172.21.15.77:0/3001982867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec815aa60 con 0x7fdec8145590 2024-04-25T01:22:12.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.612+0000 7fdebeffd640 1 -- 172.21.15.77:0/3001982867 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80039e0 con 0x7fdec8145590 2024-04-25T01:22:12.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.612+0000 7fded069e640 1 -- 172.21.15.77:0/3001982867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec8158a40 con 0x7fdec8145590 2024-04-25T01:22:12.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.613+0000 7fded069e640 1 -- 172.21.15.77:0/3001982867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec8068fb0 con 0x7fdec8145590 2024-04-25T01:22:12.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.613+0000 7fdebeffd640 1 -- 172.21.15.77:0/3001982867 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdeb801a070 con 0x7fdec8145590 2024-04-25T01:22:12.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.613+0000 7fdebeffd640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdeac06be60 0x7fdeac06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.613+0000 7fdebeffd640 1 -- 172.21.15.77:0/3001982867 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdeb808a560 con 0x7fdec8145590 2024-04-25T01:22:12.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.613+0000 7fdece43a640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdeac06be60 0x7fdeac06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.614+0000 7fdece43a640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdeac06be60 0x7fdeac06e320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdec4034000 tx=0x7fdec402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:12.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.618+0000 7fdebeffd640 1 -- 172.21.15.77:0/3001982867 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdeb80584f0 con 0x7fdec8145590 2024-04-25T01:22:12.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.787+0000 7fded069e640 1 -- 172.21.15.77:0/3001982867 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdec806ea50 con 0x7fdeac06be60 2024-04-25T01:22:12.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.795+0000 7fdebeffd640 1 -- 172.21.15.77:0/3001982867 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7fdec806ea50 con 0x7fdeac06be60 2024-04-25T01:22:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:12.780 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:22:12.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 -- 172.21.15.77:0/3001982867 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdeac06be60 msgr2=0x7fdeac06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdeac06be60 0x7fdeac06e320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdec4034000 tx=0x7fdec402e000 comp rx=0 tx=0).stop 2024-04-25T01:22:12.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 -- 172.21.15.77:0/3001982867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdec8145590 msgr2=0x7fdec8156830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdec8145590 0x7fdec8156830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdeb800c410 tx=0x7fdeb800c8d0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 -- 172.21.15.77:0/3001982867 shutdown_connections 2024-04-25T01:22:12.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdeac06be60 0x7fdeac06e320 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdec8145590 0x7fdec8156830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec815f9b0 0x7fdec815d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 --2- 172.21.15.77:0/3001982867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec806e280 0x7fdec815a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 -- 172.21.15.77:0/3001982867 >> 172.21.15.77:0/3001982867 conn(0x7fdec80b1820 msgr2=0x7fdec80b0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 -- 172.21.15.77:0/3001982867 shutdown_connections 2024-04-25T01:22:12.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.798+0000 7fded069e640 1 -- 172.21.15.77:0/3001982867 wait complete. 2024-04-25T01:22:12.790 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-25T01:22:11.161505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219400,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755320,"statfs":{"total":773094113280,"available":772869447680,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003202"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471974+0000","last_change":"2024-04-25T01:21:45.082729+0000","last_active":"2024-04-25T01:21:51.471974+0000","last_peered":"2024-04-25T01:21:51.471974+0000","last_clean":"2024-04-25T01:21:51.471974+0000","last_became_active":"2024-04-25T01:21:45.082441+0000","last_became_peered":"2024-04-25T01:21:45.082441+0000","last_unstale":"2024-04-25T01:21:51.471974+0000","last_undegraded":"2024-04-25T01:21:51.471974+0000","last_fullsized":"2024-04-25T01:21:51.471974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:31.228997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472236+0000","last_change":"2024-04-25T01:21:46.093885+0000","last_active":"2024-04-25T01:21:51.472236+0000","last_peered":"2024-04-25T01:21:51.472236+0000","last_clean":"2024-04-25T01:21:51.472236+0000","last_became_active":"2024-04-25T01:21:46.092142+0000","last_became_peered":"2024-04-25T01:21:46.092142+0000","last_unstale":"2024-04-25T01:21:51.472236+0000","last_undegraded":"2024-04-25T01:21:51.472236+0000","last_fullsized":"2024-04-25T01:21:51.472236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471738+0000","last_change":"2024-04-25T01:21:50.187217+0000","last_active":"2024-04-25T01:21:51.471738+0000","last_peered":"2024-04-25T01:21:51.471738+0000","last_clean":"2024-04-25T01:21:51.471738+0000","last_became_active":"2024-04-25T01:21:50.186698+0000","last_became_peered":"2024-04-25T01:21:50.186698+0000","last_unstale":"2024-04-25T01:21:51.471738+0000","last_undegraded":"2024-04-25T01:21:51.471738+0000","last_fullsized":"2024-04-25T01:21:51.471738+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:50.627648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472373+0000","last_change":"2024-04-25T01:21:45.078191+0000","last_active":"2024-04-25T01:21:51.472373+0000","last_peered":"2024-04-25T01:21:51.472373+0000","last_clean":"2024-04-25T01:21:51.472373+0000","last_became_active":"2024-04-25T01:21:45.077949+0000","last_became_peered":"2024-04-25T01:21:45.077949+0000","last_unstale":"2024-04-25T01:21:51.472373+0000","last_undegraded":"2024-04-25T01:21:51.472373+0000","last_fullsized":"2024-04-25T01:21:51.472373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:23.371789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:46.091354+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:46.090827+0000","last_became_peered":"2024-04-25T01:21:46.090827+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.117971+0000","last_change":"2024-04-25T01:21:50.126246+0000","last_active":"2024-04-25T01:21:51.117971+0000","last_peered":"2024-04-25T01:21:51.117971+0000","last_clean":"2024-04-25T01:21:51.117971+0000","last_became_active":"2024-04-25T01:21:50.125823+0000","last_became_peered":"2024-04-25T01:21:50.125823+0000","last_unstale":"2024-04-25T01:21:51.117971+0000","last_undegraded":"2024-04-25T01:21:51.117971+0000","last_fullsized":"2024-04-25T01:21:51.117971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:05.469540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472877+0000","last_change":"2024-04-25T01:21:45.080811+0000","last_active":"2024-04-25T01:21:51.472877+0000","last_peered":"2024-04-25T01:21:51.472877+0000","last_clean":"2024-04-25T01:21:51.472877+0000","last_became_active":"2024-04-25T01:21:45.079902+0000","last_became_peered":"2024-04-25T01:21:45.079902+0000","last_unstale":"2024-04-25T01:21:51.472877+0000","last_undegraded":"2024-04-25T01:21:51.472877+0000","last_fullsized":"2024-04-25T01:21:51.472877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:15.208981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472902+0000","last_change":"2024-04-25T01:21:46.090986+0000","last_active":"2024-04-25T01:21:51.472902+0000","last_peered":"2024-04-25T01:21:51.472902+0000","last_clean":"2024-04-25T01:21:51.472902+0000","last_became_active":"2024-04-25T01:21:46.090715+0000","last_became_peered":"2024-04-25T01:21:46.090715+0000","last_unstale":"2024-04-25T01:21:51.472902+0000","last_undegraded":"2024-04-25T01:21:51.472902+0000","last_fullsized":"2024-04-25T01:21:51.472902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:27.583834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118940+0000","last_change":"2024-04-25T01:21:50.128507+0000","last_active":"2024-04-25T01:21:51.118940+0000","last_peered":"2024-04-25T01:21:51.118940+0000","last_clean":"2024-04-25T01:21:51.118940+0000","last_became_active":"2024-04-25T01:21:50.128116+0000","last_became_peered":"2024-04-25T01:21:50.128116+0000","last_unstale":"2024-04-25T01:21:51.118940+0000","last_undegraded":"2024-04-25T01:21:51.118940+0000","last_fullsized":"2024-04-25T01:21:51.118940+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.077404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472303+0000","last_change":"2024-04-25T01:21:45.080401+0000","last_active":"2024-04-25T01:21:51.472303+0000","last_peered":"2024-04-25T01:21:51.472303+0000","last_clean":"2024-04-25T01:21:51.472303+0000","last_became_active":"2024-04-25T01:21:45.079961+0000","last_became_peered":"2024-04-25T01:21:45.079961+0000","last_unstale":"2024-04-25T01:21:51.472303+0000","last_undegraded":"2024-04-25T01:21:51.472303+0000","last_fullsized":"2024-04-25T01:21:51.472303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473268+0000","last_change":"2024-04-25T01:21:46.090524+0000","last_active":"2024-04-25T01:21:51.473268+0000","last_peered":"2024-04-25T01:21:51.473268+0000","last_clean":"2024-04-25T01:21:51.473268+0000","last_became_active":"2024-04-25T01:21:46.089869+0000","last_became_peered":"2024-04-25T01:21:46.089869+0000","last_unstale":"2024-04-25T01:21:51.473268+0000","last_undegraded":"2024-04-25T01:21:51.473268+0000","last_fullsized":"2024-04-25T01:21:51.473268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:24.344315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118039+0000","last_change":"2024-04-25T01:21:50.118075+0000","last_active":"2024-04-25T01:21:51.118039+0000","last_peered":"2024-04-25T01:21:51.118039+0000","last_clean":"2024-04-25T01:21:51.118039+0000","last_became_active":"2024-04-25T01:21:50.117657+0000","last_became_peered":"2024-04-25T01:21:50.117657+0000","last_unstale":"2024-04-25T01:21:51.118039+0000","last_undegraded":"2024-04-25T01:21:51.118039+0000","last_fullsized":"2024-04-25T01:21:51.118039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:29.510529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118231+0000","last_change":"2024-04-25T01:21:46.093011+0000","last_active":"2024-04-25T01:21:51.118231+0000","last_peered":"2024-04-25T01:21:51.118231+0000","last_clean":"2024-04-25T01:21:51.118231+0000","last_became_active":"2024-04-25T01:21:46.092710+0000","last_became_peered":"2024-04-25T01:21:46.092710+0000","last_unstale":"2024-04-25T01:21:51.118231+0000","last_undegraded":"2024-04-25T01:21:51.118231+0000","last_fullsized":"2024-04-25T01:21:51.118231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472603+0000","last_change":"2024-04-25T01:21:45.080010+0000","last_active":"2024-04-25T01:21:51.472603+0000","last_peered":"2024-04-25T01:21:51.472603+0000","last_clean":"2024-04-25T01:21:51.472603+0000","last_became_active":"2024-04-25T01:21:45.079732+0000","last_became_peered":"2024-04-25T01:21:45.079732+0000","last_unstale":"2024-04-25T01:21:51.472603+0000","last_undegraded":"2024-04-25T01:21:51.472603+0000","last_fullsized":"2024-04-25T01:21:51.472603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119330+0000","last_change":"2024-04-25T01:21:50.121929+0000","last_active":"2024-04-25T01:21:51.119330+0000","last_peered":"2024-04-25T01:21:51.119330+0000","last_clean":"2024-04-25T01:21:51.119330+0000","last_became_active":"2024-04-25T01:21:50.121288+0000","last_became_peered":"2024-04-25T01:21:50.121288+0000","last_unstale":"2024-04-25T01:21:51.119330+0000","last_undegraded":"2024-04-25T01:21:51.119330+0000","last_fullsized":"2024-04-25T01:21:51.119330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473407+0000","last_change":"2024-04-25T01:21:46.082691+0000","last_active":"2024-04-25T01:21:51.473407+0000","last_peered":"2024-04-25T01:21:51.473407+0000","last_clean":"2024-04-25T01:21:51.473407+0000","last_became_active":"2024-04-25T01:21:46.082266+0000","last_became_peered":"2024-04-25T01:21:46.082266+0000","last_unstale":"2024-04-25T01:21:51.473407+0000","last_undegraded":"2024-04-25T01:21:51.473407+0000","last_fullsized":"2024-04-25T01:21:51.473407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:22.971333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473925+0000","last_change":"2024-04-25T01:21:45.076031+0000","last_active":"2024-04-25T01:21:51.473925+0000","last_peered":"2024-04-25T01:21:51.473925+0000","last_clean":"2024-04-25T01:21:51.473925+0000","last_became_active":"2024-04-25T01:21:45.075769+0000","last_became_peered":"2024-04-25T01:21:45.075769+0000","last_unstale":"2024-04-25T01:21:51.473925+0000","last_undegraded":"2024-04-25T01:21:51.473925+0000","last_fullsized":"2024-04-25T01:21:51.473925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118699+0000","last_change":"2024-04-25T01:21:50.122266+0000","last_active":"2024-04-25T01:21:51.118699+0000","last_peered":"2024-04-25T01:21:51.118699+0000","last_clean":"2024-04-25T01:21:51.118699+0000","last_became_active":"2024-04-25T01:21:50.121879+0000","last_became_peered":"2024-04-25T01:21:50.121879+0000","last_unstale":"2024-04-25T01:21:51.118699+0000","last_undegraded":"2024-04-25T01:21:51.118699+0000","last_fullsized":"2024-04-25T01:21:51.118699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:51.825101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471956+0000","last_change":"2024-04-25T01:21:46.093175+0000","last_active":"2024-04-25T01:21:51.471956+0000","last_peered":"2024-04-25T01:21:51.471956+0000","last_clean":"2024-04-25T01:21:51.471956+0000","last_became_active":"2024-04-25T01:21:46.092327+0000","last_became_peered":"2024-04-25T01:21:46.092327+0000","last_unstale":"2024-04-25T01:21:51.471956+0000","last_undegraded":"2024-04-25T01:21:51.471956+0000","last_fullsized":"2024-04-25T01:21:51.471956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473162+0000","last_change":"2024-04-25T01:21:45.077148+0000","last_active":"2024-04-25T01:21:51.473162+0000","last_peered":"2024-04-25T01:21:51.473162+0000","last_clean":"2024-04-25T01:21:51.473162+0000","last_became_active":"2024-04-25T01:21:45.076858+0000","last_became_peered":"2024-04-25T01:21:45.076858+0000","last_unstale":"2024-04-25T01:21:51.473162+0000","last_undegraded":"2024-04-25T01:21:51.473162+0000","last_fullsized":"2024-04-25T01:21:51.473162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:15.208981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473795+0000","last_change":"2024-04-25T01:21:50.182249+0000","last_active":"2024-04-25T01:21:51.473795+0000","last_peered":"2024-04-25T01:21:51.473795+0000","last_clean":"2024-04-25T01:21:51.473795+0000","last_became_active":"2024-04-25T01:21:50.181860+0000","last_became_peered":"2024-04-25T01:21:50.181860+0000","last_unstale":"2024-04-25T01:21:51.473795+0000","last_undegraded":"2024-04-25T01:21:51.473795+0000","last_fullsized":"2024-04-25T01:21:51.473795+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:38.806941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119159+0000","last_change":"2024-04-25T01:21:46.085766+0000","last_active":"2024-04-25T01:21:51.119159+0000","last_peered":"2024-04-25T01:21:51.119159+0000","last_clean":"2024-04-25T01:21:51.119159+0000","last_became_active":"2024-04-25T01:21:46.085479+0000","last_became_peered":"2024-04-25T01:21:46.085479+0000","last_unstale":"2024-04-25T01:21:51.119159+0000","last_undegraded":"2024-04-25T01:21:51.119159+0000","last_fullsized":"2024-04-25T01:21:51.119159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:02.493076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:45.085085+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:45.084796+0000","last_became_peered":"2024-04-25T01:21:45.084796+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:25.950316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118843+0000","last_change":"2024-04-25T01:21:50.123379+0000","last_active":"2024-04-25T01:21:51.118843+0000","last_peered":"2024-04-25T01:21:51.118843+0000","last_clean":"2024-04-25T01:21:51.118843+0000","last_became_active":"2024-04-25T01:21:50.123057+0000","last_became_peered":"2024-04-25T01:21:50.123057+0000","last_unstale":"2024-04-25T01:21:51.118843+0000","last_undegraded":"2024-04-25T01:21:51.118843+0000","last_fullsized":"2024-04-25T01:21:51.118843+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:13.828811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472820+0000","last_change":"2024-04-25T01:21:46.090075+0000","last_active":"2024-04-25T01:21:51.472820+0000","last_peered":"2024-04-25T01:21:51.472820+0000","last_clean":"2024-04-25T01:21:51.472820+0000","last_became_active":"2024-04-25T01:21:46.089800+0000","last_became_peered":"2024-04-25T01:21:46.089800+0000","last_unstale":"2024-04-25T01:21:51.472820+0000","last_undegraded":"2024-04-25T01:21:51.472820+0000","last_fullsized":"2024-04-25T01:21:51.472820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119254+0000","last_change":"2024-04-25T01:21:45.080102+0000","last_active":"2024-04-25T01:21:51.119254+0000","last_peered":"2024-04-25T01:21:51.119254+0000","last_clean":"2024-04-25T01:21:51.119254+0000","last_became_active":"2024-04-25T01:21:45.079786+0000","last_became_peered":"2024-04-25T01:21:45.079786+0000","last_unstale":"2024-04-25T01:21:51.119254+0000","last_undegraded":"2024-04-25T01:21:51.119254+0000","last_fullsized":"2024-04-25T01:21:51.119254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:51.235948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118619+0000","last_change":"2024-04-25T01:21:50.185727+0000","last_active":"2024-04-25T01:21:51.118619+0000","last_peered":"2024-04-25T01:21:51.118619+0000","last_clean":"2024-04-25T01:21:51.118619+0000","last_became_active":"2024-04-25T01:21:50.184395+0000","last_became_peered":"2024-04-25T01:21:50.184395+0000","last_unstale":"2024-04-25T01:21:51.118619+0000","last_undegraded":"2024-04-25T01:21:51.118619+0000","last_fullsized":"2024-04-25T01:21:51.118619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:05.257957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472627+0000","last_change":"2024-04-25T01:21:46.099799+0000","last_active":"2024-04-25T01:21:51.472627+0000","last_peered":"2024-04-25T01:21:51.472627+0000","last_clean":"2024-04-25T01:21:51.472627+0000","last_became_active":"2024-04-25T01:21:46.099506+0000","last_became_peered":"2024-04-25T01:21:46.099506+0000","last_unstale":"2024-04-25T01:21:51.472627+0000","last_undegraded":"2024-04-25T01:21:51.472627+0000","last_fullsized":"2024-04-25T01:21:51.472627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473591+0000","last_change":"2024-04-25T01:21:45.079966+0000","last_active":"2024-04-25T01:21:51.473591+0000","last_peered":"2024-04-25T01:21:51.473591+0000","last_clean":"2024-04-25T01:21:51.473591+0000","last_became_active":"2024-04-25T01:21:45.079689+0000","last_became_peered":"2024-04-25T01:21:45.079689+0000","last_unstale":"2024-04-25T01:21:51.473591+0000","last_undegraded":"2024-04-25T01:21:51.473591+0000","last_fullsized":"2024-04-25T01:21:51.473591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471654+0000","last_change":"2024-04-25T01:21:50.184865+0000","last_active":"2024-04-25T01:21:51.471654+0000","last_peered":"2024-04-25T01:21:51.471654+0000","last_clean":"2024-04-25T01:21:51.471654+0000","last_became_active":"2024-04-25T01:21:50.184466+0000","last_became_peered":"2024-04-25T01:21:50.184466+0000","last_unstale":"2024-04-25T01:21:51.471654+0000","last_undegraded":"2024-04-25T01:21:51.471654+0000","last_fullsized":"2024-04-25T01:21:51.471654+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:42.606728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473876+0000","last_change":"2024-04-25T01:21:46.100915+0000","last_active":"2024-04-25T01:21:51.473876+0000","last_peered":"2024-04-25T01:21:51.473876+0000","last_clean":"2024-04-25T01:21:51.473876+0000","last_became_active":"2024-04-25T01:21:46.100641+0000","last_became_peered":"2024-04-25T01:21:46.100641+0000","last_unstale":"2024-04-25T01:21:51.473876+0000","last_undegraded":"2024-04-25T01:21:51.473876+0000","last_fullsized":"2024-04-25T01:21:51.473876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472835+0000","last_change":"2024-04-25T01:21:45.085103+0000","last_active":"2024-04-25T01:21:51.472835+0000","last_peered":"2024-04-25T01:21:51.472835+0000","last_clean":"2024-04-25T01:21:51.472835+0000","last_became_active":"2024-04-25T01:21:45.084805+0000","last_became_peered":"2024-04-25T01:21:45.084805+0000","last_unstale":"2024-04-25T01:21:51.472835+0000","last_undegraded":"2024-04-25T01:21:51.472835+0000","last_fullsized":"2024-04-25T01:21:51.472835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:56.444547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471942+0000","last_change":"2024-04-25T01:21:50.184815+0000","last_active":"2024-04-25T01:21:51.471942+0000","last_peered":"2024-04-25T01:21:51.471942+0000","last_clean":"2024-04-25T01:21:51.471942+0000","last_became_active":"2024-04-25T01:21:50.184441+0000","last_became_peered":"2024-04-25T01:21:50.184441+0000","last_unstale":"2024-04-25T01:21:51.471942+0000","last_undegraded":"2024-04-25T01:21:51.471942+0000","last_fullsized":"2024-04-25T01:21:51.471942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473364+0000","last_change":"2024-04-25T01:21:46.096428+0000","last_active":"2024-04-25T01:21:51.473364+0000","last_peered":"2024-04-25T01:21:51.473364+0000","last_clean":"2024-04-25T01:21:51.473364+0000","last_became_active":"2024-04-25T01:21:46.096151+0000","last_became_peered":"2024-04-25T01:21:46.096151+0000","last_unstale":"2024-04-25T01:21:51.473364+0000","last_undegraded":"2024-04-25T01:21:51.473364+0000","last_fullsized":"2024-04-25T01:21:51.473364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:42.961329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473960+0000","last_change":"2024-04-25T01:21:45.080814+0000","last_active":"2024-04-25T01:21:51.473960+0000","last_peered":"2024-04-25T01:21:51.473960+0000","last_clean":"2024-04-25T01:21:51.473960+0000","last_became_active":"2024-04-25T01:21:45.080505+0000","last_became_peered":"2024-04-25T01:21:45.080505+0000","last_unstale":"2024-04-25T01:21:51.473960+0000","last_undegraded":"2024-04-25T01:21:51.473960+0000","last_fullsized":"2024-04-25T01:21:51.473960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473549+0000","last_change":"2024-04-25T01:21:50.129319+0000","last_active":"2024-04-25T01:21:51.473549+0000","last_peered":"2024-04-25T01:21:51.473549+0000","last_clean":"2024-04-25T01:21:51.473549+0000","last_became_active":"2024-04-25T01:21:50.128797+0000","last_became_peered":"2024-04-25T01:21:50.128797+0000","last_unstale":"2024-04-25T01:21:51.473549+0000","last_undegraded":"2024-04-25T01:21:51.473549+0000","last_fullsized":"2024-04-25T01:21:51.473549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:31.623179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119209+0000","last_change":"2024-04-25T01:21:46.079764+0000","last_active":"2024-04-25T01:21:51.119209+0000","last_peered":"2024-04-25T01:21:51.119209+0000","last_clean":"2024-04-25T01:21:51.119209+0000","last_became_active":"2024-04-25T01:21:46.079475+0000","last_became_peered":"2024-04-25T01:21:46.079475+0000","last_unstale":"2024-04-25T01:21:51.119209+0000","last_undegraded":"2024-04-25T01:21:51.119209+0000","last_fullsized":"2024-04-25T01:21:51.119209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:21.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473306+0000","last_change":"2024-04-25T01:21:45.075554+0000","last_active":"2024-04-25T01:21:51.473306+0000","last_peered":"2024-04-25T01:21:51.473306+0000","last_clean":"2024-04-25T01:21:51.473306+0000","last_became_active":"2024-04-25T01:21:45.075256+0000","last_became_peered":"2024-04-25T01:21:45.075256+0000","last_unstale":"2024-04-25T01:21:51.473306+0000","last_undegraded":"2024-04-25T01:21:51.473306+0000","last_fullsized":"2024-04-25T01:21:51.473306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473829+0000","last_change":"2024-04-25T01:21:50.131544+0000","last_active":"2024-04-25T01:21:51.473829+0000","last_peered":"2024-04-25T01:21:51.473829+0000","last_clean":"2024-04-25T01:21:51.473829+0000","last_became_active":"2024-04-25T01:21:50.131125+0000","last_became_peered":"2024-04-25T01:21:50.131125+0000","last_unstale":"2024-04-25T01:21:51.473829+0000","last_undegraded":"2024-04-25T01:21:51.473829+0000","last_fullsized":"2024-04-25T01:21:51.473829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:00.042952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472863+0000","last_change":"2024-04-25T01:21:46.094965+0000","last_active":"2024-04-25T01:21:51.472863+0000","last_peered":"2024-04-25T01:21:51.472863+0000","last_clean":"2024-04-25T01:21:51.472863+0000","last_became_active":"2024-04-25T01:21:46.094515+0000","last_became_peered":"2024-04-25T01:21:46.094515+0000","last_unstale":"2024-04-25T01:21:51.472863+0000","last_undegraded":"2024-04-25T01:21:51.472863+0000","last_fullsized":"2024-04-25T01:21:51.472863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119246+0000","last_change":"2024-04-25T01:21:45.075901+0000","last_active":"2024-04-25T01:21:51.119246+0000","last_peered":"2024-04-25T01:21:51.119246+0000","last_clean":"2024-04-25T01:21:51.119246+0000","last_became_active":"2024-04-25T01:21:45.075635+0000","last_became_peered":"2024-04-25T01:21:45.075635+0000","last_unstale":"2024-04-25T01:21:51.119246+0000","last_undegraded":"2024-04-25T01:21:51.119246+0000","last_fullsized":"2024-04-25T01:21:51.119246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473778+0000","last_change":"2024-04-25T01:21:50.129249+0000","last_active":"2024-04-25T01:21:51.473778+0000","last_peered":"2024-04-25T01:21:51.473778+0000","last_clean":"2024-04-25T01:21:51.473778+0000","last_became_active":"2024-04-25T01:21:50.128867+0000","last_became_peered":"2024-04-25T01:21:50.128867+0000","last_unstale":"2024-04-25T01:21:51.473778+0000","last_undegraded":"2024-04-25T01:21:51.473778+0000","last_fullsized":"2024-04-25T01:21:51.473778+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:50.627648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471590+0000","last_change":"2024-04-25T01:21:46.078894+0000","last_active":"2024-04-25T01:21:51.471590+0000","last_peered":"2024-04-25T01:21:51.471590+0000","last_clean":"2024-04-25T01:21:51.471590+0000","last_became_active":"2024-04-25T01:21:46.078598+0000","last_became_peered":"2024-04-25T01:21:46.078598+0000","last_unstale":"2024-04-25T01:21:51.471590+0000","last_undegraded":"2024-04-25T01:21:51.471590+0000","last_fullsized":"2024-04-25T01:21:51.471590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:57.449234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471807+0000","last_change":"2024-04-25T01:21:45.080835+0000","last_active":"2024-04-25T01:21:51.471807+0000","last_peered":"2024-04-25T01:21:51.471807+0000","last_clean":"2024-04-25T01:21:51.471807+0000","last_became_active":"2024-04-25T01:21:45.080544+0000","last_became_peered":"2024-04-25T01:21:45.080544+0000","last_unstale":"2024-04-25T01:21:51.471807+0000","last_undegraded":"2024-04-25T01:21:51.471807+0000","last_fullsized":"2024-04-25T01:21:51.471807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472752+0000","last_change":"2024-04-25T01:21:50.182264+0000","last_active":"2024-04-25T01:21:51.472752+0000","last_peered":"2024-04-25T01:21:51.472752+0000","last_clean":"2024-04-25T01:21:51.472752+0000","last_became_active":"2024-04-25T01:21:50.181854+0000","last_became_peered":"2024-04-25T01:21:50.181854+0000","last_unstale":"2024-04-25T01:21:51.472752+0000","last_undegraded":"2024-04-25T01:21:51.472752+0000","last_fullsized":"2024-04-25T01:21:51.472752+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:28.383661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471748+0000","last_change":"2024-04-25T01:21:46.102152+0000","last_active":"2024-04-25T01:21:51.471748+0000","last_peered":"2024-04-25T01:21:51.471748+0000","last_clean":"2024-04-25T01:21:51.471748+0000","last_became_active":"2024-04-25T01:21:46.101917+0000","last_became_peered":"2024-04-25T01:21:46.101917+0000","last_unstale":"2024-04-25T01:21:51.471748+0000","last_undegraded":"2024-04-25T01:21:51.471748+0000","last_fullsized":"2024-04-25T01:21:51.471748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473111+0000","last_change":"2024-04-25T01:21:45.070971+0000","last_active":"2024-04-25T01:21:51.473111+0000","last_peered":"2024-04-25T01:21:51.473111+0000","last_clean":"2024-04-25T01:21:51.473111+0000","last_became_active":"2024-04-25T01:21:45.070626+0000","last_became_peered":"2024-04-25T01:21:45.070626+0000","last_unstale":"2024-04-25T01:21:51.473111+0000","last_undegraded":"2024-04-25T01:21:51.473111+0000","last_fullsized":"2024-04-25T01:21:51.473111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":177,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471866+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:21:51.471866+0000","last_peered":"2024-04-25T01:21:51.471866+0000","last_clean":"2024-04-25T01:21:51.471866+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:21:51.471866+0000","last_undegraded":"2024-04-25T01:21:51.471866+0000","last_fullsized":"2024-04-25T01:21:51.471866+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118918+0000","last_change":"2024-04-25T01:21:50.185591+0000","last_active":"2024-04-25T01:21:51.118918+0000","last_peered":"2024-04-25T01:21:51.118918+0000","last_clean":"2024-04-25T01:21:51.118918+0000","last_became_active":"2024-04-25T01:21:50.185206+0000","last_became_peered":"2024-04-25T01:21:50.185206+0000","last_unstale":"2024-04-25T01:21:51.118918+0000","last_undegraded":"2024-04-25T01:21:51.118918+0000","last_fullsized":"2024-04-25T01:21:51.118918+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.077404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472462+0000","last_change":"2024-04-25T01:21:46.104388+0000","last_active":"2024-04-25T01:21:51.472462+0000","last_peered":"2024-04-25T01:21:51.472462+0000","last_clean":"2024-04-25T01:21:51.472462+0000","last_became_active":"2024-04-25T01:21:46.103864+0000","last_became_peered":"2024-04-25T01:21:46.103864+0000","last_unstale":"2024-04-25T01:21:51.472462+0000","last_undegraded":"2024-04-25T01:21:51.472462+0000","last_fullsized":"2024-04-25T01:21:51.472462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:46.038059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119490+0000","last_change":"2024-04-25T01:21:45.078985+0000","last_active":"2024-04-25T01:21:51.119490+0000","last_peered":"2024-04-25T01:21:51.119490+0000","last_clean":"2024-04-25T01:21:51.119490+0000","last_became_active":"2024-04-25T01:21:45.078782+0000","last_became_peered":"2024-04-25T01:21:45.078782+0000","last_unstale":"2024-04-25T01:21:51.119490+0000","last_undegraded":"2024-04-25T01:21:51.119490+0000","last_fullsized":"2024-04-25T01:21:51.119490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:51.235948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118175+0000","last_change":"2024-04-25T01:21:50.182278+0000","last_active":"2024-04-25T01:21:51.118175+0000","last_peered":"2024-04-25T01:21:51.118175+0000","last_clean":"2024-04-25T01:21:51.118175+0000","last_became_active":"2024-04-25T01:21:50.181838+0000","last_became_peered":"2024-04-25T01:21:50.181838+0000","last_unstale":"2024-04-25T01:21:51.118175+0000","last_undegraded":"2024-04-25T01:21:51.118175+0000","last_fullsized":"2024-04-25T01:21:51.118175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:13.828811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473013+0000","last_change":"2024-04-25T01:21:46.096433+0000","last_active":"2024-04-25T01:21:51.473013+0000","last_peered":"2024-04-25T01:21:51.473013+0000","last_clean":"2024-04-25T01:21:51.473013+0000","last_became_active":"2024-04-25T01:21:46.096197+0000","last_became_peered":"2024-04-25T01:21:46.096197+0000","last_unstale":"2024-04-25T01:21:51.473013+0000","last_undegraded":"2024-04-25T01:21:51.473013+0000","last_fullsized":"2024-04-25T01:21:51.473013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:38.567383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473546+0000","last_change":"2024-04-25T01:21:45.072096+0000","last_active":"2024-04-25T01:21:51.473546+0000","last_peered":"2024-04-25T01:21:51.473546+0000","last_clean":"2024-04-25T01:21:51.473546+0000","last_became_active":"2024-04-25T01:21:45.071811+0000","last_became_peered":"2024-04-25T01:21:45.071811+0000","last_unstale":"2024-04-25T01:21:51.473546+0000","last_undegraded":"2024-04-25T01:21:51.473546+0000","last_fullsized":"2024-04-25T01:21:51.473546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:18.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472183+0000","last_change":"2024-04-25T01:21:50.123158+0000","last_active":"2024-04-25T01:21:51.472183+0000","last_peered":"2024-04-25T01:21:51.472183+0000","last_clean":"2024-04-25T01:21:51.472183+0000","last_became_active":"2024-04-25T01:21:50.122275+0000","last_became_peered":"2024-04-25T01:21:50.122275+0000","last_unstale":"2024-04-25T01:21:51.472183+0000","last_undegraded":"2024-04-25T01:21:51.472183+0000","last_fullsized":"2024-04-25T01:21:51.472183+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:49.064038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472847+0000","last_change":"2024-04-25T01:21:46.105159+0000","last_active":"2024-04-25T01:21:51.472847+0000","last_peered":"2024-04-25T01:21:51.472847+0000","last_clean":"2024-04-25T01:21:51.472847+0000","last_became_active":"2024-04-25T01:21:46.104876+0000","last_became_peered":"2024-04-25T01:21:46.104876+0000","last_unstale":"2024-04-25T01:21:51.472847+0000","last_undegraded":"2024-04-25T01:21:51.472847+0000","last_fullsized":"2024-04-25T01:21:51.472847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:38.567383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119570+0000","last_change":"2024-04-25T01:21:45.076578+0000","last_active":"2024-04-25T01:21:51.119570+0000","last_peered":"2024-04-25T01:21:51.119570+0000","last_clean":"2024-04-25T01:21:51.119570+0000","last_became_active":"2024-04-25T01:21:45.076022+0000","last_became_peered":"2024-04-25T01:21:45.076022+0000","last_unstale":"2024-04-25T01:21:51.119570+0000","last_undegraded":"2024-04-25T01:21:51.119570+0000","last_fullsized":"2024-04-25T01:21:51.119570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471952+0000","last_change":"2024-04-25T01:21:50.182219+0000","last_active":"2024-04-25T01:21:51.471952+0000","last_peered":"2024-04-25T01:21:51.471952+0000","last_clean":"2024-04-25T01:21:51.471952+0000","last_became_active":"2024-04-25T01:21:50.181820+0000","last_became_peered":"2024-04-25T01:21:50.181820+0000","last_unstale":"2024-04-25T01:21:51.471952+0000","last_undegraded":"2024-04-25T01:21:51.471952+0000","last_fullsized":"2024-04-25T01:21:51.471952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:20.747339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472586+0000","last_change":"2024-04-25T01:21:46.100017+0000","last_active":"2024-04-25T01:21:51.472586+0000","last_peered":"2024-04-25T01:21:51.472586+0000","last_clean":"2024-04-25T01:21:51.472586+0000","last_became_active":"2024-04-25T01:21:46.099745+0000","last_became_peered":"2024-04-25T01:21:46.099745+0000","last_unstale":"2024-04-25T01:21:51.472586+0000","last_undegraded":"2024-04-25T01:21:51.472586+0000","last_fullsized":"2024-04-25T01:21:51.472586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:30.138160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118932+0000","last_change":"2024-04-25T01:21:45.081659+0000","last_active":"2024-04-25T01:21:51.118932+0000","last_peered":"2024-04-25T01:21:51.118932+0000","last_clean":"2024-04-25T01:21:51.118932+0000","last_became_active":"2024-04-25T01:21:45.081320+0000","last_became_peered":"2024-04-25T01:21:45.081320+0000","last_unstale":"2024-04-25T01:21:51.118932+0000","last_undegraded":"2024-04-25T01:21:51.118932+0000","last_fullsized":"2024-04-25T01:21:51.118932+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:11:25.817031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118175+0000","last_change":"2024-04-25T01:21:50.122840+0000","last_active":"2024-04-25T01:21:51.118175+0000","last_peered":"2024-04-25T01:21:51.118175+0000","last_clean":"2024-04-25T01:21:51.118175+0000","last_became_active":"2024-04-25T01:21:50.122184+0000","last_became_peered":"2024-04-25T01:21:50.122184+0000","last_unstale":"2024-04-25T01:21:51.118175+0000","last_undegraded":"2024-04-25T01:21:51.118175+0000","last_fullsized":"2024-04-25T01:21:51.118175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:27.010679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472223+0000","last_change":"2024-04-25T01:21:46.105988+0000","last_active":"2024-04-25T01:21:51.472223+0000","last_peered":"2024-04-25T01:21:51.472223+0000","last_clean":"2024-04-25T01:21:51.472223+0000","last_became_active":"2024-04-25T01:21:46.105479+0000","last_became_peered":"2024-04-25T01:21:46.105479+0000","last_unstale":"2024-04-25T01:21:51.472223+0000","last_undegraded":"2024-04-25T01:21:51.472223+0000","last_fullsized":"2024-04-25T01:21:51.472223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:45.024693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119108+0000","last_change":"2024-04-25T01:21:45.074194+0000","last_active":"2024-04-25T01:21:51.119108+0000","last_peered":"2024-04-25T01:21:51.119108+0000","last_clean":"2024-04-25T01:21:51.119108+0000","last_became_active":"2024-04-25T01:21:45.073923+0000","last_became_peered":"2024-04-25T01:21:45.073923+0000","last_unstale":"2024-04-25T01:21:51.119108+0000","last_undegraded":"2024-04-25T01:21:51.119108+0000","last_fullsized":"2024-04-25T01:21:51.119108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472699+0000","last_change":"2024-04-25T01:21:50.126685+0000","last_active":"2024-04-25T01:21:51.472699+0000","last_peered":"2024-04-25T01:21:51.472699+0000","last_clean":"2024-04-25T01:21:51.472699+0000","last_became_active":"2024-04-25T01:21:50.126183+0000","last_became_peered":"2024-04-25T01:21:50.126183+0000","last_unstale":"2024-04-25T01:21:51.472699+0000","last_undegraded":"2024-04-25T01:21:51.472699+0000","last_fullsized":"2024-04-25T01:21:51.472699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.177506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118469+0000","last_change":"2024-04-25T01:21:46.096531+0000","last_active":"2024-04-25T01:21:51.118469+0000","last_peered":"2024-04-25T01:21:51.118469+0000","last_clean":"2024-04-25T01:21:51.118469+0000","last_became_active":"2024-04-25T01:21:46.096250+0000","last_became_peered":"2024-04-25T01:21:46.096250+0000","last_unstale":"2024-04-25T01:21:51.118469+0000","last_undegraded":"2024-04-25T01:21:51.118469+0000","last_fullsized":"2024-04-25T01:21:51.118469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:55.076328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.117985+0000","last_change":"2024-04-25T01:21:45.079574+0000","last_active":"2024-04-25T01:21:51.117985+0000","last_peered":"2024-04-25T01:21:51.117985+0000","last_clean":"2024-04-25T01:21:51.117985+0000","last_became_active":"2024-04-25T01:21:45.079306+0000","last_became_peered":"2024-04-25T01:21:45.079306+0000","last_unstale":"2024-04-25T01:21:51.117985+0000","last_undegraded":"2024-04-25T01:21:51.117985+0000","last_fullsized":"2024-04-25T01:21:51.117985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.488389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472488+0000","last_change":"2024-04-25T01:21:50.187252+0000","last_active":"2024-04-25T01:21:51.472488+0000","last_peered":"2024-04-25T01:21:51.472488+0000","last_clean":"2024-04-25T01:21:51.472488+0000","last_became_active":"2024-04-25T01:21:50.186509+0000","last_became_peered":"2024-04-25T01:21:50.186509+0000","last_unstale":"2024-04-25T01:21:51.472488+0000","last_undegraded":"2024-04-25T01:21:51.472488+0000","last_fullsized":"2024-04-25T01:21:51.472488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:15.374624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472403+0000","last_change":"2024-04-25T01:21:46.104318+0000","last_active":"2024-04-25T01:21:51.472403+0000","last_peered":"2024-04-25T01:21:51.472403+0000","last_clean":"2024-04-25T01:21:51.472403+0000","last_became_active":"2024-04-25T01:21:46.103937+0000","last_became_peered":"2024-04-25T01:21:46.103937+0000","last_unstale":"2024-04-25T01:21:51.472403+0000","last_undegraded":"2024-04-25T01:21:51.472403+0000","last_fullsized":"2024-04-25T01:21:51.472403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:27.802903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473263+0000","last_change":"2024-04-25T01:21:45.076486+0000","last_active":"2024-04-25T01:21:51.473263+0000","last_peered":"2024-04-25T01:21:51.473263+0000","last_clean":"2024-04-25T01:21:51.473263+0000","last_became_active":"2024-04-25T01:21:45.076164+0000","last_became_peered":"2024-04-25T01:21:45.076164+0000","last_unstale":"2024-04-25T01:21:51.473263+0000","last_undegraded":"2024-04-25T01:21:51.473263+0000","last_fullsized":"2024-04-25T01:21:51.473263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:21.050998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472905+0000","last_change":"2024-04-25T01:21:50.182316+0000","last_active":"2024-04-25T01:21:51.472905+0000","last_peered":"2024-04-25T01:21:51.472905+0000","last_clean":"2024-04-25T01:21:51.472905+0000","last_became_active":"2024-04-25T01:21:50.181918+0000","last_became_peered":"2024-04-25T01:21:50.181918+0000","last_unstale":"2024-04-25T01:21:51.472905+0000","last_undegraded":"2024-04-25T01:21:51.472905+0000","last_fullsized":"2024-04-25T01:21:51.472905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:37.041783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472461+0000","last_change":"2024-04-25T01:21:46.100833+0000","last_active":"2024-04-25T01:21:51.472461+0000","last_peered":"2024-04-25T01:21:51.472461+0000","last_clean":"2024-04-25T01:21:51.472461+0000","last_became_active":"2024-04-25T01:21:46.100591+0000","last_became_peered":"2024-04-25T01:21:46.100591+0000","last_unstale":"2024-04-25T01:21:51.472461+0000","last_undegraded":"2024-04-25T01:21:51.472461+0000","last_fullsized":"2024-04-25T01:21:51.472461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:36.537071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472415+0000","last_change":"2024-04-25T01:21:45.078469+0000","last_active":"2024-04-25T01:21:51.472415+0000","last_peered":"2024-04-25T01:21:51.472415+0000","last_clean":"2024-04-25T01:21:51.472415+0000","last_became_active":"2024-04-25T01:21:45.078231+0000","last_became_peered":"2024-04-25T01:21:45.078231+0000","last_unstale":"2024-04-25T01:21:51.472415+0000","last_undegraded":"2024-04-25T01:21:51.472415+0000","last_fullsized":"2024-04-25T01:21:51.472415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:23.371789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472786+0000","last_change":"2024-04-25T01:21:50.185207+0000","last_active":"2024-04-25T01:21:51.472786+0000","last_peered":"2024-04-25T01:21:51.472786+0000","last_clean":"2024-04-25T01:21:51.472786+0000","last_became_active":"2024-04-25T01:21:50.184793+0000","last_became_peered":"2024-04-25T01:21:50.184793+0000","last_unstale":"2024-04-25T01:21:51.472786+0000","last_undegraded":"2024-04-25T01:21:51.472786+0000","last_fullsized":"2024-04-25T01:21:51.472786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:15.374624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472438+0000","last_change":"2024-04-25T01:21:46.104460+0000","last_active":"2024-04-25T01:21:51.472438+0000","last_peered":"2024-04-25T01:21:51.472438+0000","last_clean":"2024-04-25T01:21:51.472438+0000","last_became_active":"2024-04-25T01:21:46.104163+0000","last_became_peered":"2024-04-25T01:21:46.104163+0000","last_unstale":"2024-04-25T01:21:51.472438+0000","last_undegraded":"2024-04-25T01:21:51.472438+0000","last_fullsized":"2024-04-25T01:21:51.472438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:27.583834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:55.386967+0000","last_change":"2024-04-25T01:21:45.076242+0000","last_active":"2024-04-25T01:21:55.386967+0000","last_peered":"2024-04-25T01:21:55.386967+0000","last_clean":"2024-04-25T01:21:55.386967+0000","last_became_active":"2024-04-25T01:21:45.075952+0000","last_became_peered":"2024-04-25T01:21:45.075952+0000","last_unstale":"2024-04-25T01:21:55.386967+0000","last_undegraded":"2024-04-25T01:21:55.386967+0000","last_fullsized":"2024-04-25T01:21:55.386967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:52.481389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473757+0000","last_change":"2024-04-25T01:21:50.128837+0000","last_active":"2024-04-25T01:21:51.473757+0000","last_peered":"2024-04-25T01:21:51.473757+0000","last_clean":"2024-04-25T01:21:51.473757+0000","last_became_active":"2024-04-25T01:21:50.128462+0000","last_became_peered":"2024-04-25T01:21:50.128462+0000","last_unstale":"2024-04-25T01:21:51.473757+0000","last_undegraded":"2024-04-25T01:21:51.473757+0000","last_fullsized":"2024-04-25T01:21:51.473757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:17:42.606728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118967+0000","last_change":"2024-04-25T01:21:46.100341+0000","last_active":"2024-04-25T01:21:51.118967+0000","last_peered":"2024-04-25T01:21:51.118967+0000","last_clean":"2024-04-25T01:21:51.118967+0000","last_became_active":"2024-04-25T01:21:46.099961+0000","last_became_peered":"2024-04-25T01:21:46.099961+0000","last_unstale":"2024-04-25T01:21:51.118967+0000","last_undegraded":"2024-04-25T01:21:51.118967+0000","last_fullsized":"2024-04-25T01:21:51.118967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:15.176543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473450+0000","last_change":"2024-04-25T01:21:45.077748+0000","last_active":"2024-04-25T01:21:51.473450+0000","last_peered":"2024-04-25T01:21:51.473450+0000","last_clean":"2024-04-25T01:21:51.473450+0000","last_became_active":"2024-04-25T01:21:45.077494+0000","last_became_peered":"2024-04-25T01:21:45.077494+0000","last_unstale":"2024-04-25T01:21:51.473450+0000","last_undegraded":"2024-04-25T01:21:51.473450+0000","last_fullsized":"2024-04-25T01:21:51.473450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:25.950316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118695+0000","last_change":"2024-04-25T01:21:50.126572+0000","last_active":"2024-04-25T01:21:51.118695+0000","last_peered":"2024-04-25T01:21:51.118695+0000","last_clean":"2024-04-25T01:21:51.118695+0000","last_became_active":"2024-04-25T01:21:50.126013+0000","last_became_peered":"2024-04-25T01:21:50.126013+0000","last_unstale":"2024-04-25T01:21:51.118695+0000","last_undegraded":"2024-04-25T01:21:51.118695+0000","last_fullsized":"2024-04-25T01:21:51.118695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:19.877429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471878+0000","last_change":"2024-04-25T01:21:46.091135+0000","last_active":"2024-04-25T01:21:51.471878+0000","last_peered":"2024-04-25T01:21:51.471878+0000","last_clean":"2024-04-25T01:21:51.471878+0000","last_became_active":"2024-04-25T01:21:46.090493+0000","last_became_peered":"2024-04-25T01:21:46.090493+0000","last_unstale":"2024-04-25T01:21:51.471878+0000","last_undegraded":"2024-04-25T01:21:51.471878+0000","last_fullsized":"2024-04-25T01:21:51.471878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472529+0000","last_change":"2024-04-25T01:21:45.075179+0000","last_active":"2024-04-25T01:21:51.472529+0000","last_peered":"2024-04-25T01:21:51.472529+0000","last_clean":"2024-04-25T01:21:51.472529+0000","last_became_active":"2024-04-25T01:21:45.074903+0000","last_became_peered":"2024-04-25T01:21:45.074903+0000","last_unstale":"2024-04-25T01:21:51.472529+0000","last_undegraded":"2024-04-25T01:21:51.472529+0000","last_fullsized":"2024-04-25T01:21:51.472529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471819+0000","last_change":"2024-04-25T01:21:50.126406+0000","last_active":"2024-04-25T01:21:51.471819+0000","last_peered":"2024-04-25T01:21:51.471819+0000","last_clean":"2024-04-25T01:21:51.471819+0000","last_became_active":"2024-04-25T01:21:50.126029+0000","last_became_peered":"2024-04-25T01:21:50.126029+0000","last_unstale":"2024-04-25T01:21:51.471819+0000","last_undegraded":"2024-04-25T01:21:51.471819+0000","last_fullsized":"2024-04-25T01:21:51.471819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:38.806941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472398+0000","last_change":"2024-04-25T01:21:46.100746+0000","last_active":"2024-04-25T01:21:51.472398+0000","last_peered":"2024-04-25T01:21:51.472398+0000","last_clean":"2024-04-25T01:21:51.472398+0000","last_became_active":"2024-04-25T01:21:46.100475+0000","last_became_peered":"2024-04-25T01:21:46.100475+0000","last_unstale":"2024-04-25T01:21:51.472398+0000","last_undegraded":"2024-04-25T01:21:51.472398+0000","last_fullsized":"2024-04-25T01:21:51.472398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:27.802903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119162+0000","last_change":"2024-04-25T01:21:45.082455+0000","last_active":"2024-04-25T01:21:51.119162+0000","last_peered":"2024-04-25T01:21:51.119162+0000","last_clean":"2024-04-25T01:21:51.119162+0000","last_became_active":"2024-04-25T01:21:45.082135+0000","last_became_peered":"2024-04-25T01:21:45.082135+0000","last_unstale":"2024-04-25T01:21:51.119162+0000","last_undegraded":"2024-04-25T01:21:51.119162+0000","last_fullsized":"2024-04-25T01:21:51.119162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:06.296153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471896+0000","last_change":"2024-04-25T01:21:50.184859+0000","last_active":"2024-04-25T01:21:51.471896+0000","last_peered":"2024-04-25T01:21:51.471896+0000","last_clean":"2024-04-25T01:21:51.471896+0000","last_became_active":"2024-04-25T01:21:50.184475+0000","last_became_peered":"2024-04-25T01:21:50.184475+0000","last_unstale":"2024-04-25T01:21:51.471896+0000","last_undegraded":"2024-04-25T01:21:51.471896+0000","last_fullsized":"2024-04-25T01:21:51.471896+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:00.042952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472589+0000","last_change":"2024-04-25T01:21:50.184894+0000","last_active":"2024-04-25T01:21:51.472589+0000","last_peered":"2024-04-25T01:21:51.472589+0000","last_clean":"2024-04-25T01:21:51.472589+0000","last_became_active":"2024-04-25T01:21:50.184392+0000","last_became_peered":"2024-04-25T01:21:50.184392+0000","last_unstale":"2024-04-25T01:21:51.472589+0000","last_undegraded":"2024-04-25T01:21:51.472589+0000","last_fullsized":"2024-04-25T01:21:51.472589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:31.842249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472948+0000","last_change":"2024-04-25T01:21:46.092888+0000","last_active":"2024-04-25T01:21:51.472948+0000","last_peered":"2024-04-25T01:21:51.472948+0000","last_clean":"2024-04-25T01:21:51.472948+0000","last_became_active":"2024-04-25T01:21:46.092607+0000","last_became_peered":"2024-04-25T01:21:46.092607+0000","last_unstale":"2024-04-25T01:21:51.472948+0000","last_undegraded":"2024-04-25T01:21:51.472948+0000","last_fullsized":"2024-04-25T01:21:51.472948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:09:22.971333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472329+0000","last_change":"2024-04-25T01:21:45.070532+0000","last_active":"2024-04-25T01:21:51.472329+0000","last_peered":"2024-04-25T01:21:51.472329+0000","last_clean":"2024-04-25T01:21:51.472329+0000","last_became_active":"2024-04-25T01:21:45.070256+0000","last_became_peered":"2024-04-25T01:21:45.070256+0000","last_unstale":"2024-04-25T01:21:51.472329+0000","last_undegraded":"2024-04-25T01:21:51.472329+0000","last_fullsized":"2024-04-25T01:21:51.472329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:20.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119151+0000","last_change":"2024-04-25T01:21:50.130056+0000","last_active":"2024-04-25T01:21:51.119151+0000","last_peered":"2024-04-25T01:21:51.119151+0000","last_clean":"2024-04-25T01:21:51.119151+0000","last_became_active":"2024-04-25T01:21:50.129541+0000","last_became_peered":"2024-04-25T01:21:50.129541+0000","last_unstale":"2024-04-25T01:21:51.119151+0000","last_undegraded":"2024-04-25T01:21:51.119151+0000","last_fullsized":"2024-04-25T01:21:51.119151+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:47.000674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471871+0000","last_change":"2024-04-25T01:21:46.083728+0000","last_active":"2024-04-25T01:21:51.471871+0000","last_peered":"2024-04-25T01:21:51.471871+0000","last_clean":"2024-04-25T01:21:51.471871+0000","last_became_active":"2024-04-25T01:21:46.083458+0000","last_became_peered":"2024-04-25T01:21:46.083458+0000","last_unstale":"2024-04-25T01:21:51.471871+0000","last_undegraded":"2024-04-25T01:21:51.471871+0000","last_fullsized":"2024-04-25T01:21:51.471871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472093+0000","last_change":"2024-04-25T01:21:45.077308+0000","last_active":"2024-04-25T01:21:51.472093+0000","last_peered":"2024-04-25T01:21:51.472093+0000","last_clean":"2024-04-25T01:21:51.472093+0000","last_became_active":"2024-04-25T01:21:45.076815+0000","last_became_peered":"2024-04-25T01:21:45.076815+0000","last_unstale":"2024-04-25T01:21:51.472093+0000","last_undegraded":"2024-04-25T01:21:51.472093+0000","last_fullsized":"2024-04-25T01:21:51.472093+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:43.833096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118432+0000","last_change":"2024-04-25T01:21:45.079268+0000","last_active":"2024-04-25T01:21:51.118432+0000","last_peered":"2024-04-25T01:21:51.118432+0000","last_clean":"2024-04-25T01:21:51.118432+0000","last_became_active":"2024-04-25T01:21:45.078934+0000","last_became_peered":"2024-04-25T01:21:45.078934+0000","last_unstale":"2024-04-25T01:21:51.118432+0000","last_undegraded":"2024-04-25T01:21:51.118432+0000","last_fullsized":"2024-04-25T01:21:51.118432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:19:14.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472316+0000","last_change":"2024-04-25T01:21:46.086329+0000","last_active":"2024-04-25T01:21:51.472316+0000","last_peered":"2024-04-25T01:21:51.472316+0000","last_clean":"2024-04-25T01:21:51.472316+0000","last_became_active":"2024-04-25T01:21:46.086069+0000","last_became_peered":"2024-04-25T01:21:46.086069+0000","last_unstale":"2024-04-25T01:21:51.472316+0000","last_undegraded":"2024-04-25T01:21:51.472316+0000","last_fullsized":"2024-04-25T01:21:51.472316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119441+0000","last_change":"2024-04-25T01:21:50.122334+0000","last_active":"2024-04-25T01:21:51.119441+0000","last_peered":"2024-04-25T01:21:51.119441+0000","last_clean":"2024-04-25T01:21:51.119441+0000","last_became_active":"2024-04-25T01:21:50.121974+0000","last_became_peered":"2024-04-25T01:21:50.121974+0000","last_unstale":"2024-04-25T01:21:51.119441+0000","last_undegraded":"2024-04-25T01:21:51.119441+0000","last_fullsized":"2024-04-25T01:21:51.119441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:59.115674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473488+0000","last_change":"2024-04-25T01:21:45.082740+0000","last_active":"2024-04-25T01:21:51.473488+0000","last_peered":"2024-04-25T01:21:51.473488+0000","last_clean":"2024-04-25T01:21:51.473488+0000","last_became_active":"2024-04-25T01:21:45.082439+0000","last_became_peered":"2024-04-25T01:21:45.082439+0000","last_unstale":"2024-04-25T01:21:51.473488+0000","last_undegraded":"2024-04-25T01:21:51.473488+0000","last_fullsized":"2024-04-25T01:21:51.473488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:44.052166+0000","last_clean_scrub_stamp":"2024-04-25T01:21:44.052166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:52.481389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118237+0000","last_change":"2024-04-25T01:21:46.088483+0000","last_active":"2024-04-25T01:21:51.118237+0000","last_peered":"2024-04-25T01:21:51.118237+0000","last_clean":"2024-04-25T01:21:51.118237+0000","last_became_active":"2024-04-25T01:21:46.088113+0000","last_became_peered":"2024-04-25T01:21:46.088113+0000","last_unstale":"2024-04-25T01:21:51.118237+0000","last_undegraded":"2024-04-25T01:21:51.118237+0000","last_fullsized":"2024-04-25T01:21:51.118237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:22.055684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472316+0000","last_change":"2024-04-25T01:21:50.125319+0000","last_active":"2024-04-25T01:21:51.472316+0000","last_peered":"2024-04-25T01:21:51.472316+0000","last_clean":"2024-04-25T01:21:51.472316+0000","last_became_active":"2024-04-25T01:21:50.124926+0000","last_became_peered":"2024-04-25T01:21:50.124926+0000","last_unstale":"2024-04-25T01:21:51.472316+0000","last_undegraded":"2024-04-25T01:21:51.472316+0000","last_fullsized":"2024-04-25T01:21:51.472316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:49.096198+0000","last_clean_scrub_stamp":"2024-04-25T01:21:49.096198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:20.747339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118296+0000","last_change":"2024-04-25T01:21:46.093690+0000","last_active":"2024-04-25T01:21:51.118296+0000","last_peered":"2024-04-25T01:21:51.118296+0000","last_clean":"2024-04-25T01:21:51.118296+0000","last_became_active":"2024-04-25T01:21:46.093392+0000","last_became_peered":"2024-04-25T01:21:46.093392+0000","last_unstale":"2024-04-25T01:21:51.118296+0000","last_undegraded":"2024-04-25T01:21:51.118296+0000","last_fullsized":"2024-04-25T01:21:51.118296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119144+0000","last_change":"2024-04-25T01:21:46.089771+0000","last_active":"2024-04-25T01:21:51.119144+0000","last_peered":"2024-04-25T01:21:51.119144+0000","last_clean":"2024-04-25T01:21:51.119144+0000","last_became_active":"2024-04-25T01:21:46.089489+0000","last_became_peered":"2024-04-25T01:21:46.089489+0000","last_unstale":"2024-04-25T01:21:51.119144+0000","last_undegraded":"2024-04-25T01:21:51.119144+0000","last_fullsized":"2024-04-25T01:21:51.119144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:16.213667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473158+0000","last_change":"2024-04-25T01:21:46.099812+0000","last_active":"2024-04-25T01:21:51.473158+0000","last_peered":"2024-04-25T01:21:51.473158+0000","last_clean":"2024-04-25T01:21:51.473158+0000","last_became_active":"2024-04-25T01:21:46.099508+0000","last_became_peered":"2024-04-25T01:21:46.099508+0000","last_unstale":"2024-04-25T01:21:51.473158+0000","last_undegraded":"2024-04-25T01:21:51.473158+0000","last_fullsized":"2024-04-25T01:21:51.473158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:45.024693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472233+0000","last_change":"2024-04-25T01:21:46.083637+0000","last_active":"2024-04-25T01:21:51.472233+0000","last_peered":"2024-04-25T01:21:51.472233+0000","last_clean":"2024-04-25T01:21:51.472233+0000","last_became_active":"2024-04-25T01:21:46.083371+0000","last_became_peered":"2024-04-25T01:21:46.083371+0000","last_unstale":"2024-04-25T01:21:51.472233+0000","last_undegraded":"2024-04-25T01:21:51.472233+0000","last_fullsized":"2024-04-25T01:21:51.472233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473043+0000","last_change":"2024-04-25T01:21:46.087822+0000","last_active":"2024-04-25T01:21:51.473043+0000","last_peered":"2024-04-25T01:21:51.473043+0000","last_clean":"2024-04-25T01:21:51.473043+0000","last_became_active":"2024-04-25T01:21:46.087555+0000","last_became_peered":"2024-04-25T01:21:46.087555+0000","last_unstale":"2024-04-25T01:21:51.473043+0000","last_undegraded":"2024-04-25T01:21:51.473043+0000","last_fullsized":"2024-04-25T01:21:51.473043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473261+0000","last_change":"2024-04-25T01:21:46.092978+0000","last_active":"2024-04-25T01:21:51.473261+0000","last_peered":"2024-04-25T01:21:51.473261+0000","last_clean":"2024-04-25T01:21:51.473261+0000","last_became_active":"2024-04-25T01:21:46.092720+0000","last_became_peered":"2024-04-25T01:21:46.092720+0000","last_unstale":"2024-04-25T01:21:51.473261+0000","last_undegraded":"2024-04-25T01:21:51.473261+0000","last_fullsized":"2024-04-25T01:21:51.473261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119314+0000","last_change":"2024-04-25T01:21:46.100969+0000","last_active":"2024-04-25T01:21:51.119314+0000","last_peered":"2024-04-25T01:21:51.119314+0000","last_clean":"2024-04-25T01:21:51.119314+0000","last_became_active":"2024-04-25T01:21:46.100687+0000","last_became_peered":"2024-04-25T01:21:46.100687+0000","last_unstale":"2024-04-25T01:21:51.119314+0000","last_undegraded":"2024-04-25T01:21:51.119314+0000","last_fullsized":"2024-04-25T01:21:51.119314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:25.471184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472539+0000","last_change":"2024-04-25T01:21:46.087688+0000","last_active":"2024-04-25T01:21:51.472539+0000","last_peered":"2024-04-25T01:21:51.472539+0000","last_clean":"2024-04-25T01:21:51.472539+0000","last_became_active":"2024-04-25T01:21:46.087422+0000","last_became_peered":"2024-04-25T01:21:46.087422+0000","last_unstale":"2024-04-25T01:21:51.472539+0000","last_undegraded":"2024-04-25T01:21:51.472539+0000","last_fullsized":"2024-04-25T01:21:51.472539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472217+0000","last_change":"2024-04-25T01:21:46.096585+0000","last_active":"2024-04-25T01:21:51.472217+0000","last_peered":"2024-04-25T01:21:51.472217+0000","last_clean":"2024-04-25T01:21:51.472217+0000","last_became_active":"2024-04-25T01:21:46.096299+0000","last_became_peered":"2024-04-25T01:21:46.096299+0000","last_unstale":"2024-04-25T01:21:51.472217+0000","last_undegraded":"2024-04-25T01:21:51.472217+0000","last_fullsized":"2024-04-25T01:21:51.472217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472814+0000","last_change":"2024-04-25T01:21:46.085048+0000","last_active":"2024-04-25T01:21:51.472814+0000","last_peered":"2024-04-25T01:21:51.472814+0000","last_clean":"2024-04-25T01:21:51.472814+0000","last_became_active":"2024-04-25T01:21:46.084766+0000","last_became_peered":"2024-04-25T01:21:46.084766+0000","last_unstale":"2024-04-25T01:21:51.472814+0000","last_undegraded":"2024-04-25T01:21:51.472814+0000","last_fullsized":"2024-04-25T01:21:51.472814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:19.723609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473111+0000","last_change":"2024-04-25T01:21:46.105035+0000","last_active":"2024-04-25T01:21:51.473111+0000","last_peered":"2024-04-25T01:21:51.473111+0000","last_clean":"2024-04-25T01:21:51.473111+0000","last_became_active":"2024-04-25T01:21:46.104776+0000","last_became_peered":"2024-04-25T01:21:46.104776+0000","last_unstale":"2024-04-25T01:21:51.473111+0000","last_undegraded":"2024-04-25T01:21:51.473111+0000","last_fullsized":"2024-04-25T01:21:51.473111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471992+0000","last_change":"2024-04-25T01:21:46.100414+0000","last_active":"2024-04-25T01:21:51.471992+0000","last_peered":"2024-04-25T01:21:51.471992+0000","last_clean":"2024-04-25T01:21:51.471992+0000","last_became_active":"2024-04-25T01:21:46.100140+0000","last_became_peered":"2024-04-25T01:21:46.100140+0000","last_unstale":"2024-04-25T01:21:51.471992+0000","last_undegraded":"2024-04-25T01:21:51.471992+0000","last_fullsized":"2024-04-25T01:21:51.471992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:55.076328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118784+0000","last_change":"2024-04-25T01:21:46.092761+0000","last_active":"2024-04-25T01:21:51.118784+0000","last_peered":"2024-04-25T01:21:51.118784+0000","last_clean":"2024-04-25T01:21:51.118784+0000","last_became_active":"2024-04-25T01:21:46.092230+0000","last_became_peered":"2024-04-25T01:21:46.092230+0000","last_unstale":"2024-04-25T01:21:51.118784+0000","last_undegraded":"2024-04-25T01:21:51.118784+0000","last_fullsized":"2024-04-25T01:21:51.118784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472724+0000","last_change":"2024-04-25T01:21:46.104951+0000","last_active":"2024-04-25T01:21:51.472724+0000","last_peered":"2024-04-25T01:21:51.472724+0000","last_clean":"2024-04-25T01:21:51.472724+0000","last_became_active":"2024-04-25T01:21:46.104662+0000","last_became_peered":"2024-04-25T01:21:46.104662+0000","last_unstale":"2024-04-25T01:21:51.472724+0000","last_undegraded":"2024-04-25T01:21:51.472724+0000","last_fullsized":"2024-04-25T01:21:51.472724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:46.588303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119527+0000","last_change":"2024-04-25T01:21:46.091860+0000","last_active":"2024-04-25T01:21:51.119527+0000","last_peered":"2024-04-25T01:21:51.119527+0000","last_clean":"2024-04-25T01:21:51.119527+0000","last_became_active":"2024-04-25T01:21:46.091569+0000","last_became_peered":"2024-04-25T01:21:46.091569+0000","last_unstale":"2024-04-25T01:21:51.119527+0000","last_undegraded":"2024-04-25T01:21:51.119527+0000","last_fullsized":"2024-04-25T01:21:51.119527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:57.449234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473437+0000","last_change":"2024-04-25T01:21:46.099956+0000","last_active":"2024-04-25T01:21:51.473437+0000","last_peered":"2024-04-25T01:21:51.473437+0000","last_clean":"2024-04-25T01:21:51.473437+0000","last_became_active":"2024-04-25T01:21:46.099687+0000","last_became_peered":"2024-04-25T01:21:46.099687+0000","last_unstale":"2024-04-25T01:21:51.473437+0000","last_undegraded":"2024-04-25T01:21:51.473437+0000","last_fullsized":"2024-04-25T01:21:51.473437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:30.138160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472108+0000","last_change":"2024-04-25T01:21:46.102084+0000","last_active":"2024-04-25T01:21:51.472108+0000","last_peered":"2024-04-25T01:21:51.472108+0000","last_clean":"2024-04-25T01:21:51.472108+0000","last_became_active":"2024-04-25T01:21:46.101539+0000","last_became_peered":"2024-04-25T01:21:46.101539+0000","last_unstale":"2024-04-25T01:21:51.472108+0000","last_undegraded":"2024-04-25T01:21:51.472108+0000","last_fullsized":"2024-04-25T01:21:51.472108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:52:01.218611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473063+0000","last_change":"2024-04-25T01:21:46.089149+0000","last_active":"2024-04-25T01:21:51.473063+0000","last_peered":"2024-04-25T01:21:51.473063+0000","last_clean":"2024-04-25T01:21:51.473063+0000","last_became_active":"2024-04-25T01:21:46.088845+0000","last_became_peered":"2024-04-25T01:21:46.088845+0000","last_unstale":"2024-04-25T01:21:51.473063+0000","last_undegraded":"2024-04-25T01:21:51.473063+0000","last_fullsized":"2024-04-25T01:21:51.473063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472391+0000","last_change":"2024-04-25T01:21:46.087764+0000","last_active":"2024-04-25T01:21:51.472391+0000","last_peered":"2024-04-25T01:21:51.472391+0000","last_clean":"2024-04-25T01:21:51.472391+0000","last_became_active":"2024-04-25T01:21:46.087481+0000","last_became_peered":"2024-04-25T01:21:46.087481+0000","last_unstale":"2024-04-25T01:21:51.472391+0000","last_undegraded":"2024-04-25T01:21:51.472391+0000","last_fullsized":"2024-04-25T01:21:51.472391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:16.213667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473322+0000","last_change":"2024-04-25T01:21:46.093981+0000","last_active":"2024-04-25T01:21:51.473322+0000","last_peered":"2024-04-25T01:21:51.473322+0000","last_clean":"2024-04-25T01:21:51.473322+0000","last_became_active":"2024-04-25T01:21:46.093729+0000","last_became_peered":"2024-04-25T01:21:46.093729+0000","last_unstale":"2024-04-25T01:21:51.473322+0000","last_undegraded":"2024-04-25T01:21:51.473322+0000","last_fullsized":"2024-04-25T01:21:51.473322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118866+0000","last_change":"2024-04-25T01:21:46.094258+0000","last_active":"2024-04-25T01:21:51.118866+0000","last_peered":"2024-04-25T01:21:51.118866+0000","last_clean":"2024-04-25T01:21:51.118866+0000","last_became_active":"2024-04-25T01:21:46.093633+0000","last_became_peered":"2024-04-25T01:21:46.093633+0000","last_unstale":"2024-04-25T01:21:51.118866+0000","last_undegraded":"2024-04-25T01:21:51.118866+0000","last_fullsized":"2024-04-25T01:21:51.118866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:52.240635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.118920+0000","last_change":"2024-04-25T01:21:46.093764+0000","last_active":"2024-04-25T01:21:51.118920+0000","last_peered":"2024-04-25T01:21:51.118920+0000","last_clean":"2024-04-25T01:21:51.118920+0000","last_became_active":"2024-04-25T01:21:46.093468+0000","last_became_peered":"2024-04-25T01:21:46.093468+0000","last_unstale":"2024-04-25T01:21:51.118920+0000","last_undegraded":"2024-04-25T01:21:51.118920+0000","last_fullsized":"2024-04-25T01:21:51.118920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:51.902077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.119290+0000","last_change":"2024-04-25T01:21:46.096039+0000","last_active":"2024-04-25T01:21:51.119290+0000","last_peered":"2024-04-25T01:21:51.119290+0000","last_clean":"2024-04-25T01:21:51.119290+0000","last_became_active":"2024-04-25T01:21:46.095773+0000","last_became_peered":"2024-04-25T01:21:46.095773+0000","last_unstale":"2024-04-25T01:21:51.119290+0000","last_undegraded":"2024-04-25T01:21:51.119290+0000","last_fullsized":"2024-04-25T01:21:51.119290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:01.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472805+0000","last_change":"2024-04-25T01:21:46.106320+0000","last_active":"2024-04-25T01:21:51.472805+0000","last_peered":"2024-04-25T01:21:51.472805+0000","last_clean":"2024-04-25T01:21:51.472805+0000","last_became_active":"2024-04-25T01:21:46.106064+0000","last_became_peered":"2024-04-25T01:21:46.106064+0000","last_unstale":"2024-04-25T01:21:51.472805+0000","last_undegraded":"2024-04-25T01:21:51.472805+0000","last_fullsized":"2024-04-25T01:21:51.472805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:24.344315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.471632+0000","last_change":"2024-04-25T01:21:46.093521+0000","last_active":"2024-04-25T01:21:51.471632+0000","last_peered":"2024-04-25T01:21:51.471632+0000","last_clean":"2024-04-25T01:21:51.471632+0000","last_became_active":"2024-04-25T01:21:46.093269+0000","last_became_peered":"2024-04-25T01:21:46.093269+0000","last_unstale":"2024-04-25T01:21:51.471632+0000","last_undegraded":"2024-04-25T01:21:51.471632+0000","last_fullsized":"2024-04-25T01:21:51.471632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:27.281314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472027+0000","last_change":"2024-04-25T01:21:46.102728+0000","last_active":"2024-04-25T01:21:51.472027+0000","last_peered":"2024-04-25T01:21:51.472027+0000","last_clean":"2024-04-25T01:21:51.472027+0000","last_became_active":"2024-04-25T01:21:46.102475+0000","last_became_peered":"2024-04-25T01:21:46.102475+0000","last_unstale":"2024-04-25T01:21:51.472027+0000","last_undegraded":"2024-04-25T01:21:51.472027+0000","last_fullsized":"2024-04-25T01:21:51.472027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472990+0000","last_change":"2024-04-25T01:21:46.101690+0000","last_active":"2024-04-25T01:21:51.472990+0000","last_peered":"2024-04-25T01:21:51.472990+0000","last_clean":"2024-04-25T01:21:51.472990+0000","last_became_active":"2024-04-25T01:21:46.101404+0000","last_became_peered":"2024-04-25T01:21:46.101404+0000","last_unstale":"2024-04-25T01:21:51.472990+0000","last_undegraded":"2024-04-25T01:21:51.472990+0000","last_fullsized":"2024-04-25T01:21:51.472990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:15.838084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472132+0000","last_change":"2024-04-25T01:21:46.103900+0000","last_active":"2024-04-25T01:21:51.472132+0000","last_peered":"2024-04-25T01:21:51.472132+0000","last_clean":"2024-04-25T01:21:51.472132+0000","last_became_active":"2024-04-25T01:21:46.103633+0000","last_became_peered":"2024-04-25T01:21:46.103633+0000","last_unstale":"2024-04-25T01:21:51.472132+0000","last_undegraded":"2024-04-25T01:21:51.472132+0000","last_fullsized":"2024-04-25T01:21:51.472132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:42.961329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472786+0000","last_change":"2024-04-25T01:21:46.102312+0000","last_active":"2024-04-25T01:21:51.472786+0000","last_peered":"2024-04-25T01:21:51.472786+0000","last_clean":"2024-04-25T01:21:51.472786+0000","last_became_active":"2024-04-25T01:21:46.101113+0000","last_became_peered":"2024-04-25T01:21:46.101113+0000","last_unstale":"2024-04-25T01:21:51.472786+0000","last_undegraded":"2024-04-25T01:21:51.472786+0000","last_fullsized":"2024-04-25T01:21:51.472786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:46.038059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473192+0000","last_change":"2024-04-25T01:21:46.098593+0000","last_active":"2024-04-25T01:21:51.473192+0000","last_peered":"2024-04-25T01:21:51.473192+0000","last_clean":"2024-04-25T01:21:51.473192+0000","last_became_active":"2024-04-25T01:21:46.098289+0000","last_became_peered":"2024-04-25T01:21:46.098289+0000","last_unstale":"2024-04-25T01:21:51.473192+0000","last_undegraded":"2024-04-25T01:21:51.473192+0000","last_fullsized":"2024-04-25T01:21:51.473192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:56.723961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.473504+0000","last_change":"2024-04-25T01:21:46.093954+0000","last_active":"2024-04-25T01:21:51.473504+0000","last_peered":"2024-04-25T01:21:51.473504+0000","last_clean":"2024-04-25T01:21:51.473504+0000","last_became_active":"2024-04-25T01:21:46.093658+0000","last_became_peered":"2024-04-25T01:21:46.093658+0000","last_unstale":"2024-04-25T01:21:51.473504+0000","last_undegraded":"2024-04-25T01:21:51.473504+0000","last_fullsized":"2024-04-25T01:21:51.473504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:54:09.789465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T01:21:51.472718+0000","last_change":"2024-04-25T01:21:46.108321+0000","last_active":"2024-04-25T01:21:51.472718+0000","last_peered":"2024-04-25T01:21:51.472718+0000","last_clean":"2024-04-25T01:21:51.472718+0000","last_became_active":"2024-04-25T01:21:46.108056+0000","last_became_peered":"2024-04-25T01:21:46.108056+0000","last_unstale":"2024-04-25T01:21:51.472718+0000","last_undegraded":"2024-04-25T01:21:51.472718+0000","last_fullsized":"2024-04-25T01:21:51.472718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:45.056852+0000","last_clean_scrub_stamp":"2024-04-25T01:21:45.056852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:47.785755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:22:12.795 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T01:22:12.795 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T01:22:12.795 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T01:22:12.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T01:22:12.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.946+0000 7f5a8cd70640 1 Processor -- start 2024-04-25T01:22:12.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.946+0000 7f5a8cd70640 1 -- start start 2024-04-25T01:22:12.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a8cd70640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5a88145540 0x7f5a88160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a8cd70640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5a880d0610 0x7f5a8806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a8cd70640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a881608a0 con 0x7f5a880ce010 2024-04-25T01:22:12.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a8cd70640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f5a880ce4b0 con 0x7f5a880d0610 2024-04-25T01:22:12.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a8cd70640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5a880ce790 con 0x7f5a88145540 2024-04-25T01:22:12.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a86d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5a880d0610 0x7f5a8806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a85d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5a880ce010 0x7f5a880cff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56578/0 (socket says 172.21.15.77:56578) 2024-04-25T01:22:12.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a85d9b640 1 -- 172.21.15.77:0/1787999854 learned_addr learned my addr 172.21.15.77:0/1787999854 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:12.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a8659c640 1 --2- 172.21.15.77:0/1787999854 >> v2:172.21.15.121:3300/0 conn(0x7f5a88145540 0x7f5a88160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a86d9d640 1 -- 172.21.15.77:0/1787999854 >> v2:172.21.15.121:3300/0 conn(0x7f5a88145540 msgr2=0x7f5a88160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:12.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a86d9d640 1 --2- 172.21.15.77:0/1787999854 >> v2:172.21.15.121:3300/0 conn(0x7f5a88145540 0x7f5a88160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a86d9d640 1 -- 172.21.15.77:0/1787999854 >> v1:172.21.15.77:6789/0 conn(0x7f5a880ce010 legacy=0x7f5a880cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a86d9d640 1 -- 172.21.15.77:0/1787999854 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a880cea70 con 0x7f5a880d0610 2024-04-25T01:22:12.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a8559a640 1 -- 172.21.15.77:0/1787999854 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 223373173 0 0) 0x7f5a78001110 con 0x7f5a880ce010 2024-04-25T01:22:12.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.947+0000 7f5a8659c640 1 --2- 172.21.15.77:0/1787999854 >> v2:172.21.15.121:3300/0 conn(0x7f5a88145540 0x7f5a88160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:12.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.948+0000 7f5a86d9d640 1 --2- 172.21.15.77:0/1787999854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a880d0610 0x7f5a8806c990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5a88072360 tx=0x7f5a8002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad64481e6fe7335d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:12.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.948+0000 7f5a8559a640 1 -- 172.21.15.77:0/1787999854 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a80007740 con 0x7f5a880d0610 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.948+0000 7f5a8559a640 1 -- 172.21.15.77:0/1787999854 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a800078e0 con 0x7f5a880d0610 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.948+0000 7f5a8559a640 1 -- 172.21.15.77:0/1787999854 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a80007cb0 con 0x7f5a880d0610 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.948+0000 7f5a8cd70640 1 -- 172.21.15.77:0/1787999854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a880d0610 msgr2=0x7f5a8806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.948+0000 7f5a8cd70640 1 --2- 172.21.15.77:0/1787999854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a880d0610 0x7f5a8806c990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5a88072360 tx=0x7f5a8002a5f0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.948+0000 7f5a8cd70640 1 -- 172.21.15.77:0/1787999854 shutdown_connections 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.948+0000 7f5a8cd70640 1 --2- 172.21.15.77:0/1787999854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a880d0610 0x7f5a8806c990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.948+0000 7f5a8cd70640 1 --2- 172.21.15.77:0/1787999854 >> v2:172.21.15.121:3300/0 conn(0x7f5a88145540 0x7f5a88160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.948+0000 7f5a8cd70640 1 -- 172.21.15.77:0/1787999854 >> 172.21.15.77:0/1787999854 conn(0x7f5a8806b540 msgr2=0x7f5a8806b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.949+0000 7f5a8cd70640 1 -- 172.21.15.77:0/1787999854 shutdown_connections 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.949+0000 7f5a8cd70640 1 -- 172.21.15.77:0/1787999854 wait complete. 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.950+0000 7f5a8cd70640 1 Processor -- start 2024-04-25T01:22:12.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.950+0000 7f5a8cd70640 1 -- start start 2024-04-25T01:22:12.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a8cd70640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a880ce010 0x7f5a881578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a8cd70640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a880d0610 0x7f5a88143d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a8cd70640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a88145540 0x7f5a88158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a8cd70640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5a8806ecd0 con 0x7f5a880d0610 2024-04-25T01:22:12.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a8cd70640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5a880cf030 con 0x7f5a88145540 2024-04-25T01:22:12.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a8cd70640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5a88145920 con 0x7f5a880ce010 2024-04-25T01:22:12.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a85d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a880d0610 0x7f5a88143d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a85d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a880d0610 0x7f5a88143d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56586/0 (socket says 172.21.15.77:56586) 2024-04-25T01:22:12.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a85d9b640 1 -- 172.21.15.77:0/2550926817 learned_addr learned my addr 172.21.15.77:0/2550926817 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a85d9b640 1 -- 172.21.15.77:0/2550926817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a880ce010 msgr2=0x7f5a881578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a86d9d640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a88145540 0x7f5a88158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a85d9b640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a880ce010 0x7f5a881578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a85d9b640 1 -- 172.21.15.77:0/2550926817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a88145540 msgr2=0x7f5a88158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a85d9b640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a88145540 0x7f5a88158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a85d9b640 1 -- 172.21.15.77:0/2550926817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a881f5990 con 0x7f5a880d0610 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a85d9b640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a880d0610 0x7f5a88143d50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800c7b0 tx=0x7f5a7800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a76ffd640 1 -- 172.21.15.77:0/2550926817 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a78018070 con 0x7f5a880d0610 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a76ffd640 1 -- 172.21.15.77:0/2550926817 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a780072d0 con 0x7f5a880d0610 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a8cd70640 1 -- 172.21.15.77:0/2550926817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a881f5c70 con 0x7f5a880d0610 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a76ffd640 1 -- 172.21.15.77:0/2550926817 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a780047e0 con 0x7f5a880d0610 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.951+0000 7f5a8cd70640 1 -- 172.21.15.77:0/2550926817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a881f6230 con 0x7f5a880d0610 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.952+0000 7f5a8cd70640 1 -- 172.21.15.77:0/2550926817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a88068fb0 con 0x7f5a880d0610 2024-04-25T01:22:12.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.953+0000 7f5a76ffd640 1 -- 172.21.15.77:0/2550926817 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5a7801b070 con 0x7f5a880d0610 2024-04-25T01:22:12.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.953+0000 7f5a76ffd640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5a6806be30 0x7f5a6806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:12.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.953+0000 7f5a76ffd640 1 -- 172.21.15.77:0/2550926817 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5a7808a470 con 0x7f5a880d0610 2024-04-25T01:22:12.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.953+0000 7f5a8659c640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5a6806be30 0x7f5a6806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:12.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.954+0000 7f5a8659c640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5a6806be30 0x7f5a6806e2f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c006fd0 tx=0x7f5a7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:12.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:12.957+0000 7f5a76ffd640 1 -- 172.21.15.77:0/2550926817 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5a881608a0 con 0x7f5a880d0610 2024-04-25T01:22:13.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.165+0000 7f5a8cd70640 1 -- 172.21.15.77:0/2550926817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5a88072a90 con 0x7f5a880d0610 2024-04-25T01:22:13.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.165+0000 7f5a76ffd640 1 -- 172.21.15.77:0/2550926817 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5a78058b70 con 0x7f5a880d0610 2024-04-25T01:22:13.146 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:13.146 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T01:22:13.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 -- 172.21.15.77:0/2550926817 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5a6806be30 msgr2=0x7f5a6806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5a6806be30 0x7f5a6806e2f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c006fd0 tx=0x7f5a7c008040 comp rx=0 tx=0).stop 2024-04-25T01:22:13.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 -- 172.21.15.77:0/2550926817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a880d0610 msgr2=0x7f5a88143d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a880d0610 0x7f5a88143d50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800c7b0 tx=0x7f5a7800cc70 comp rx=0 tx=0).stop 2024-04-25T01:22:13.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 -- 172.21.15.77:0/2550926817 shutdown_connections 2024-04-25T01:22:13.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5a6806be30 0x7f5a6806e2f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a880d0610 0x7f5a88143d50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a88145540 0x7f5a88158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 --2- 172.21.15.77:0/2550926817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a880ce010 0x7f5a881578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 -- 172.21.15.77:0/2550926817 >> 172.21.15.77:0/2550926817 conn(0x7f5a8806b540 msgr2=0x7f5a8815b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:13.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 -- 172.21.15.77:0/2550926817 shutdown_connections 2024-04-25T01:22:13.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.167+0000 7f5a8cd70640 1 -- 172.21.15.77:0/2550926817 wait complete. 2024-04-25T01:22:13.159 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T01:22:13.159 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:13.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.276+0000 7ff0d7b77640 1 Processor -- start 2024-04-25T01:22:13.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.276+0000 7ff0d7b77640 1 -- start start 2024-04-25T01:22:13.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.277+0000 7ff0d7b77640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 0x7ff0d006e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.277+0000 7ff0d7b77640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0d015f400 0x7ff0d006eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.277+0000 7ff0d7b77640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d0070e60 con 0x7ff0d0143d20 2024-04-25T01:22:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.277+0000 7ff0d7b77640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff0d0071140 con 0x7ff0d015f400 2024-04-25T01:22:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.277+0000 7ff0d7b77640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d0071420 con 0x7ff0d0143940 2024-04-25T01:22:13.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.277+0000 7ff0d5913640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 0x7ff0d006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.277+0000 7ff0d6114640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0d015f400 0x7ff0d006eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.277+0000 7ff0d5913640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 0x7ff0d006e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:34060/0 (socket says 172.21.15.77:34060) 2024-04-25T01:22:13.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.277+0000 7ff0d6114640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0d015f400 0x7ff0d006eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:49018/0 (socket says 172.21.15.77:49018) 2024-04-25T01:22:13.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.277+0000 7ff0d5913640 1 -- 172.21.15.77:0/673020505 learned_addr learned my addr 172.21.15.77:0/673020505 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:13.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.277+0000 7ff0d5112640 1 --1- 172.21.15.77:0/673020505 >> v1:172.21.15.121:6789/0 conn(0x7ff0d0143d20 0x7ff0d015ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:59702/0 (socket says 172.21.15.77:59702) 2024-04-25T01:22:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.278+0000 7ff0d5913640 1 -- 172.21.15.77:0/673020505 >> v2:172.21.15.121:3300/0 conn(0x7ff0d015f400 msgr2=0x7ff0d006eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.278+0000 7ff0d5913640 1 --2- 172.21.15.77:0/673020505 >> v2:172.21.15.121:3300/0 conn(0x7ff0d015f400 0x7ff0d006eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.278+0000 7ff0d5913640 1 -- 172.21.15.77:0/673020505 >> v1:172.21.15.121:6789/0 conn(0x7ff0d0143d20 legacy=0x7ff0d015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.278+0000 7ff0d5913640 1 -- 172.21.15.77:0/673020505 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d0071700 con 0x7ff0d0143940 2024-04-25T01:22:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.278+0000 7ff0d6114640 1 --2- 172.21.15.77:0/673020505 >> v2:172.21.15.121:3300/0 conn(0x7ff0d015f400 0x7ff0d006eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.278+0000 7ff0d5913640 1 --2- 172.21.15.77:0/673020505 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 0x7ff0d006e650 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009870 tx=0x7ff0cc02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2624007572d1cd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.279+0000 7ff0d4911640 1 -- 172.21.15.77:0/673020505 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc037070 con 0x7ff0d0143940 2024-04-25T01:22:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.279+0000 7ff0d4911640 1 -- 172.21.15.77:0/673020505 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0cc032440 con 0x7ff0d0143940 2024-04-25T01:22:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.279+0000 7ff0d4911640 1 -- 172.21.15.77:0/673020505 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc032810 con 0x7ff0d0143940 2024-04-25T01:22:13.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.279+0000 7ff0d7b77640 1 -- 172.21.15.77:0/673020505 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 msgr2=0x7ff0d006e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.279+0000 7ff0d7b77640 1 --2- 172.21.15.77:0/673020505 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 0x7ff0d006e650 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009870 tx=0x7ff0cc02a270 comp rx=0 tx=0).stop 2024-04-25T01:22:13.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.279+0000 7ff0d7b77640 1 -- 172.21.15.77:0/673020505 shutdown_connections 2024-04-25T01:22:13.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.279+0000 7ff0d7b77640 1 --2- 172.21.15.77:0/673020505 >> v2:172.21.15.121:3300/0 conn(0x7ff0d015f400 0x7ff0d006eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.279+0000 7ff0d7b77640 1 --2- 172.21.15.77:0/673020505 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 0x7ff0d006e650 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.279+0000 7ff0d7b77640 1 -- 172.21.15.77:0/673020505 >> 172.21.15.77:0/673020505 conn(0x7ff0d00b1820 msgr2=0x7ff0d00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.279+0000 7ff0d7b77640 1 -- 172.21.15.77:0/673020505 shutdown_connections 2024-04-25T01:22:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.279+0000 7ff0d7b77640 1 -- 172.21.15.77:0/673020505 wait complete. 2024-04-25T01:22:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.280+0000 7ff0d7b77640 1 Processor -- start 2024-04-25T01:22:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d7b77640 1 -- start start 2024-04-25T01:22:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d7b77640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 0x7ff0d015b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d7b77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0d0143d20 0x7ff0d0158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d7b77640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d015f400 0x7ff0d015a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d7b77640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d0071fb0 con 0x7ff0d0143d20 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d7b77640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d0071e30 con 0x7ff0d015f400 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d7b77640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d0072130 con 0x7ff0d0143940 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d5112640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0d0143d20 0x7ff0d0158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d5112640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0d0143d20 0x7ff0d0158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56594/0 (socket says 172.21.15.77:56594) 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d5112640 1 -- 172.21.15.77:0/2127634548 learned_addr learned my addr 172.21.15.77:0/2127634548 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d6114640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d015f400 0x7ff0d015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.281+0000 7ff0d5913640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 0x7ff0d015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.282+0000 7ff0d5112640 1 -- 172.21.15.77:0/2127634548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 msgr2=0x7ff0d015b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.282+0000 7ff0d5112640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 0x7ff0d015b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.282+0000 7ff0d5112640 1 -- 172.21.15.77:0/2127634548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d015f400 msgr2=0x7ff0d015a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.282+0000 7ff0d5112640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d015f400 0x7ff0d015a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.282+0000 7ff0d5112640 1 -- 172.21.15.77:0/2127634548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d01591c0 con 0x7ff0d0143d20 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.282+0000 7ff0d5112640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0d0143d20 0x7ff0d0158970 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000c430 tx=0x7ff0c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.282+0000 7ff0c67fc640 1 -- 172.21.15.77:0/2127634548 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00074e0 con 0x7ff0d0143d20 2024-04-25T01:22:13.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.282+0000 7ff0c67fc640 1 -- 172.21.15.77:0/2127634548 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c0007680 con 0x7ff0d0143d20 2024-04-25T01:22:13.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.282+0000 7ff0c67fc640 1 -- 172.21.15.77:0/2127634548 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0004a00 con 0x7ff0d0143d20 2024-04-25T01:22:13.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.282+0000 7ff0d7b77640 1 -- 172.21.15.77:0/2127634548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d01594a0 con 0x7ff0d0143d20 2024-04-25T01:22:13.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.282+0000 7ff0d7b77640 1 -- 172.21.15.77:0/2127634548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d0159a60 con 0x7ff0d0143d20 2024-04-25T01:22:13.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.283+0000 7ff0d7b77640 1 -- 172.21.15.77:0/2127634548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d0158f10 con 0x7ff0d0143d20 2024-04-25T01:22:13.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.283+0000 7ff0c67fc640 1 -- 172.21.15.77:0/2127634548 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff0c001a020 con 0x7ff0d0143d20 2024-04-25T01:22:13.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.283+0000 7ff0c67fc640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff0b406be30 0x7ff0b406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.284+0000 7ff0c67fc640 1 -- 172.21.15.77:0/2127634548 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff0c008a9b0 con 0x7ff0d0143d20 2024-04-25T01:22:13.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.284+0000 7ff0d5913640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff0b406be30 0x7ff0b406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.284+0000 7ff0d5913640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff0b406be30 0x7ff0b406e2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc034000 tx=0x7ff0cc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.288+0000 7ff0c67fc640 1 -- 172.21.15.77:0/2127634548 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff0c0059030 con 0x7ff0d0143d20 2024-04-25T01:22:13.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.465+0000 7ff0d7b77640 1 -- 172.21.15.77:0/2127634548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0d0068fb0 con 0x7ff0d0143d20 2024-04-25T01:22:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.466+0000 7ff0c67fc640 1 -- 172.21.15.77:0/2127634548 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7ff0c0058930 con 0x7ff0d0143d20 2024-04-25T01:22:13.447 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:13.447 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:13.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 -- 172.21.15.77:0/2127634548 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff0b406be30 msgr2=0x7ff0b406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff0b406be30 0x7ff0b406e2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc034000 tx=0x7ff0cc02e000 comp rx=0 tx=0).stop 2024-04-25T01:22:13.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 -- 172.21.15.77:0/2127634548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0d0143d20 msgr2=0x7ff0d0158970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0d0143d20 0x7ff0d0158970 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000c430 tx=0x7ff0c000c8f0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 -- 172.21.15.77:0/2127634548 shutdown_connections 2024-04-25T01:22:13.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff0b406be30 0x7ff0b406e2f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0d0143d20 0x7ff0d0158970 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d015f400 0x7ff0d015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 --2- 172.21.15.77:0/2127634548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d0143940 0x7ff0d015b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 -- 172.21.15.77:0/2127634548 >> 172.21.15.77:0/2127634548 conn(0x7ff0d00b1820 msgr2=0x7ff0d00af690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:13.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 -- 172.21.15.77:0/2127634548 shutdown_connections 2024-04-25T01:22:13.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.468+0000 7ff0d7b77640 1 -- 172.21.15.77:0/2127634548 wait complete. 2024-04-25T01:22:13.460 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:13.461 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:13.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.580+0000 7fbc51ad4640 1 Processor -- start 2024-04-25T01:22:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.580+0000 7fbc51ad4640 1 -- start start 2024-04-25T01:22:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc51ad4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc4c0722e0 0x7fbc4c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc51ad4640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbc4c15fc40 0x7fbc4c1600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc51ad4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbc4c0cedc0 0x7fbc4c0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc51ad4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fbc4c160aa0 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc51ad4640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fbc4c06da40 con 0x7fbc4c15fc40 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc51ad4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4c06cec0 con 0x7fbc4c0722e0 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc4bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbc4c0cedc0 0x7fbc4c0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc4bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbc4c0cedc0 0x7fbc4c0cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56604/0 (socket says 172.21.15.77:56604) 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc4bfff640 1 -- 172.21.15.77:0/2969007396 learned_addr learned my addr 172.21.15.77:0/2969007396 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc4b7fe640 1 --2- 172.21.15.77:0/2969007396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc4c0722e0 0x7fbc4c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc4affd640 1 --2- 172.21.15.77:0/2969007396 >> v2:172.21.15.121:3300/0 conn(0x7fbc4c15fc40 0x7fbc4c1600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc4bfff640 1 -- 172.21.15.77:0/2969007396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc4c0722e0 msgr2=0x7fbc4c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc4bfff640 1 --2- 172.21.15.77:0/2969007396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc4c0722e0 0x7fbc4c06ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc4bfff640 1 -- 172.21.15.77:0/2969007396 >> v2:172.21.15.121:3300/0 conn(0x7fbc4c15fc40 msgr2=0x7fbc4c1600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc4bfff640 1 --2- 172.21.15.77:0/2969007396 >> v2:172.21.15.121:3300/0 conn(0x7fbc4c15fc40 0x7fbc4c1600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc4bfff640 1 -- 172.21.15.77:0/2969007396 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c0cf790 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.581+0000 7fbc4affd640 1 --2- 172.21.15.77:0/2969007396 >> v2:172.21.15.121:3300/0 conn(0x7fbc4c15fc40 0x7fbc4c1600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.582+0000 7fbc4bfff640 1 --2- 172.21.15.77:0/2969007396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc4c0cedc0 0x7fbc4c0cf250 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0b4f60 tx=0x7fbc4402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37de82e7034ec861 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.582+0000 7fbc4a7fc640 1 -- 172.21.15.77:0/2969007396 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc44036070 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.582+0000 7fbc4a7fc640 1 -- 172.21.15.77:0/2969007396 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc44031600 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.582+0000 7fbc4a7fc640 1 -- 172.21.15.77:0/2969007396 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc440319d0 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.582+0000 7fbc51ad4640 1 -- 172.21.15.77:0/2969007396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc4c0cedc0 msgr2=0x7fbc4c0cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.582+0000 7fbc51ad4640 1 --2- 172.21.15.77:0/2969007396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc4c0cedc0 0x7fbc4c0cf250 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0b4f60 tx=0x7fbc4402a610 comp rx=0 tx=0).stop 2024-04-25T01:22:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.582+0000 7fbc51ad4640 1 -- 172.21.15.77:0/2969007396 shutdown_connections 2024-04-25T01:22:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.582+0000 7fbc51ad4640 1 --2- 172.21.15.77:0/2969007396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc4c0cedc0 0x7fbc4c0cf250 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.582+0000 7fbc51ad4640 1 --2- 172.21.15.77:0/2969007396 >> v2:172.21.15.121:3300/0 conn(0x7fbc4c15fc40 0x7fbc4c1600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.583+0000 7fbc51ad4640 1 --2- 172.21.15.77:0/2969007396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc4c0722e0 0x7fbc4c06ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.583+0000 7fbc51ad4640 1 -- 172.21.15.77:0/2969007396 >> 172.21.15.77:0/2969007396 conn(0x7fbc4c06b720 msgr2=0x7fbc4c0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.583+0000 7fbc51ad4640 1 -- 172.21.15.77:0/2969007396 shutdown_connections 2024-04-25T01:22:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.583+0000 7fbc51ad4640 1 -- 172.21.15.77:0/2969007396 wait complete. 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.584+0000 7fbc51ad4640 1 Processor -- start 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.584+0000 7fbc51ad4640 1 -- start start 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc51ad4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc4c0722e0 0x7fbc4c159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc51ad4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc4c0cedc0 0x7fbc4c157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc51ad4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc4c15fc40 0x7fbc4c1f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc51ad4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c06d3f0 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc51ad4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c06e410 con 0x7fbc4c0722e0 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc51ad4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4c06d6d0 con 0x7fbc4c15fc40 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc4affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc4c0cedc0 0x7fbc4c157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc4affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc4c0cedc0 0x7fbc4c157af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56616/0 (socket says 172.21.15.77:56616) 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc4affd640 1 -- 172.21.15.77:0/1642351152 learned_addr learned my addr 172.21.15.77:0/1642351152 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc4bfff640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc4c15fc40 0x7fbc4c1f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc4b7fe640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc4c0722e0 0x7fbc4c159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc4affd640 1 -- 172.21.15.77:0/1642351152 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc4c15fc40 msgr2=0x7fbc4c1f4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc4affd640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc4c15fc40 0x7fbc4c1f4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc4affd640 1 -- 172.21.15.77:0/1642351152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc4c0722e0 msgr2=0x7fbc4c159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc4affd640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc4c0722e0 0x7fbc4c159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc4affd640 1 -- 172.21.15.77:0/1642351152 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c1581f0 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc4affd640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc4c0cedc0 0x7fbc4c157af0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000e7e0 tx=0x7fbc4000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc33fff640 1 -- 172.21.15.77:0/1642351152 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4000f040 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc33fff640 1 -- 172.21.15.77:0/1642351152 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc40004030 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc51ad4640 1 -- 172.21.15.77:0/1642351152 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4c1584d0 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc33fff640 1 -- 172.21.15.77:0/1642351152 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40004400 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.585+0000 7fbc51ad4640 1 -- 172.21.15.77:0/1642351152 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4c158a90 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.586+0000 7fbc51ad4640 1 -- 172.21.15.77:0/1642351152 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc4c068fb0 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.587+0000 7fbc33fff640 1 -- 172.21.15.77:0/1642351152 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbc4000c6d0 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.587+0000 7fbc33fff640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbc3806be60 0x7fbc3806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.587+0000 7fbc33fff640 1 -- 172.21.15.77:0/1642351152 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbc4008afd0 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.587+0000 7fbc4b7fe640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbc3806be60 0x7fbc3806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.588+0000 7fbc4b7fe640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbc3806be60 0x7fbc3806e320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0098f0 tx=0x7fbc3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.591+0000 7fbc33fff640 1 -- 172.21.15.77:0/1642351152 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbc40017020 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.765+0000 7fbc51ad4640 1 -- 172.21.15.77:0/1642351152 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbc4c072c90 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.765+0000 7fbc33fff640 1 -- 172.21.15.77:0/1642351152 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7fbc400595b0 con 0x7fbc4c0cedc0 2024-04-25T01:22:13.746 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:13.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 -- 172.21.15.77:0/1642351152 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbc3806be60 msgr2=0x7fbc3806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbc3806be60 0x7fbc3806e320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0098f0 tx=0x7fbc3c008040 comp rx=0 tx=0).stop 2024-04-25T01:22:13.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 -- 172.21.15.77:0/1642351152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc4c0cedc0 msgr2=0x7fbc4c157af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc4c0cedc0 0x7fbc4c157af0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000e7e0 tx=0x7fbc4000eca0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 -- 172.21.15.77:0/1642351152 shutdown_connections 2024-04-25T01:22:13.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbc3806be60 0x7fbc3806e320 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc4c0cedc0 0x7fbc4c157af0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc4c15fc40 0x7fbc4c1f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 --2- 172.21.15.77:0/1642351152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc4c0722e0 0x7fbc4c159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 -- 172.21.15.77:0/1642351152 >> 172.21.15.77:0/1642351152 conn(0x7fbc4c06b720 msgr2=0x7fbc4c15f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:13.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 -- 172.21.15.77:0/1642351152 shutdown_connections 2024-04-25T01:22:13.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.767+0000 7fbc51ad4640 1 -- 172.21.15.77:0/1642351152 wait complete. 2024-04-25T01:22:13.759 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:13.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.877+0000 7fdac0b5a640 1 Processor -- start 2024-04-25T01:22:13.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.877+0000 7fdac0b5a640 1 -- start start 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.878+0000 7fdac0b5a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdabc0b9a90 0x7fdabc0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.878+0000 7fdac0b5a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdabc0d27b0 0x7fdabc0d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.878+0000 7fdac0b5a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdabc0cea40 con 0x7fdabc0d23d0 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.878+0000 7fdac0b5a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fdabc0cece0 con 0x7fdabc0b9a90 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.878+0000 7fdac0b5a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdabc0cefc0 con 0x7fdabc0d27b0 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.879+0000 7fdab9d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdabc0b9a90 0x7fdabc0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.879+0000 7fdaba59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdabc0d23d0 0x7fdabc0b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56594/0 (socket says 172.21.15.77:56594) 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.879+0000 7fdaba59c640 1 -- 172.21.15.77:0/4036473103 learned_addr learned my addr 172.21.15.77:0/4036473103 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.879+0000 7fdabad9d640 1 --2- 172.21.15.77:0/4036473103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdabc0d27b0 0x7fdabc0d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.879+0000 7fdab9d9b640 1 -- 172.21.15.77:0/4036473103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdabc0d27b0 msgr2=0x7fdabc0d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.879+0000 7fdab9d9b640 1 --2- 172.21.15.77:0/4036473103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdabc0d27b0 0x7fdabc0d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.879+0000 7fdab9d9b640 1 -- 172.21.15.77:0/4036473103 >> v1:172.21.15.77:6789/0 conn(0x7fdabc0d23d0 legacy=0x7fdabc0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.879+0000 7fdab9d9b640 1 -- 172.21.15.77:0/4036473103 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc0cf2a0 con 0x7fdabc0b9a90 2024-04-25T01:22:13.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.879+0000 7fdab959a640 1 -- 172.21.15.77:0/4036473103 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 30846271 0 0) 0x7fdabc0cea40 con 0x7fdabc0d23d0 2024-04-25T01:22:13.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdab9d9b640 1 --2- 172.21.15.77:0/4036473103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdabc0b9a90 0x7fdabc0d4920 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdab00097a0 tx=0x7fdab002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38397c012b1359e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdab959a640 1 -- 172.21.15.77:0/4036473103 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab0037070 con 0x7fdabc0b9a90 2024-04-25T01:22:13.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdab959a640 1 -- 172.21.15.77:0/4036473103 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab0032440 con 0x7fdabc0b9a90 2024-04-25T01:22:13.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdab959a640 1 -- 172.21.15.77:0/4036473103 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab0032810 con 0x7fdabc0b9a90 2024-04-25T01:22:13.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdac0b5a640 1 -- 172.21.15.77:0/4036473103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdabc0b9a90 msgr2=0x7fdabc0d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdac0b5a640 1 --2- 172.21.15.77:0/4036473103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdabc0b9a90 0x7fdabc0d4920 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdab00097a0 tx=0x7fdab002a210 comp rx=0 tx=0).stop 2024-04-25T01:22:13.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdac0b5a640 1 -- 172.21.15.77:0/4036473103 shutdown_connections 2024-04-25T01:22:13.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdac0b5a640 1 --2- 172.21.15.77:0/4036473103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdabc0d27b0 0x7fdabc0d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdac0b5a640 1 --2- 172.21.15.77:0/4036473103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdabc0b9a90 0x7fdabc0d4920 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdac0b5a640 1 -- 172.21.15.77:0/4036473103 >> 172.21.15.77:0/4036473103 conn(0x7fdabc0d3e30 msgr2=0x7fdabc0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:13.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdac0b5a640 1 -- 172.21.15.77:0/4036473103 shutdown_connections 2024-04-25T01:22:13.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.880+0000 7fdac0b5a640 1 -- 172.21.15.77:0/4036473103 wait complete. 2024-04-25T01:22:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.882+0000 7fdac0b5a640 1 Processor -- start 2024-04-25T01:22:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.882+0000 7fdac0b5a640 1 -- start start 2024-04-25T01:22:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.882+0000 7fdac0b5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdabc0b9a90 0x7fdabc0ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.882+0000 7fdac0b5a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdabc0d23d0 0x7fdabc1f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.882+0000 7fdac0b5a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdabc0d27b0 0x7fdabc06be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.882+0000 7fdac0b5a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc0cfd80 con 0x7fdabc0b9a90 2024-04-25T01:22:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.882+0000 7fdac0b5a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc0cf860 con 0x7fdabc0d27b0 2024-04-25T01:22:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.882+0000 7fdac0b5a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdabc0b6c80 con 0x7fdabc0d23d0 2024-04-25T01:22:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.882+0000 7fdaba59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdabc0b9a90 0x7fdabc0ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.882+0000 7fdaba59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdabc0b9a90 0x7fdabc0ca3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56624/0 (socket says 172.21.15.77:56624) 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.882+0000 7fdaba59c640 1 -- 172.21.15.77:0/3224402153 learned_addr learned my addr 172.21.15.77:0/3224402153 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdabad9d640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdabc0d27b0 0x7fdabc06be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdab9d9b640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdabc0d23d0 0x7fdabc1f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdaba59c640 1 -- 172.21.15.77:0/3224402153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdabc0d23d0 msgr2=0x7fdabc1f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdaba59c640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdabc0d23d0 0x7fdabc1f7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdaba59c640 1 -- 172.21.15.77:0/3224402153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdabc0d27b0 msgr2=0x7fdabc06be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdaba59c640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdabc0d27b0 0x7fdabc06be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdaba59c640 1 -- 172.21.15.77:0/3224402153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc1f77a0 con 0x7fdabc0b9a90 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdaba59c640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdabc0b9a90 0x7fdabc0ca3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdab400c890 tx=0x7fdab400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdaaaffd640 1 -- 172.21.15.77:0/3224402153 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4018070 con 0x7fdabc0b9a90 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdaaaffd640 1 -- 172.21.15.77:0/3224402153 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab400a5e0 con 0x7fdabc0b9a90 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdac0b5a640 1 -- 172.21.15.77:0/3224402153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdabc1f7a80 con 0x7fdabc0b9a90 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdaaaffd640 1 -- 172.21.15.77:0/3224402153 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab400a9d0 con 0x7fdabc0b9a90 2024-04-25T01:22:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.883+0000 7fdac0b5a640 1 -- 172.21.15.77:0/3224402153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdabc15be90 con 0x7fdabc0b9a90 2024-04-25T01:22:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.884+0000 7fdac0b5a640 1 -- 172.21.15.77:0/3224402153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdabc1f7fa0 con 0x7fdabc0b9a90 2024-04-25T01:22:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.885+0000 7fdaaaffd640 1 -- 172.21.15.77:0/3224402153 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdab401b070 con 0x7fdabc0b9a90 2024-04-25T01:22:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.885+0000 7fdaaaffd640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fda9c06be60 0x7fda9c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.885+0000 7fdaaaffd640 1 -- 172.21.15.77:0/3224402153 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdab408a6e0 con 0x7fdabc0b9a90 2024-04-25T01:22:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.885+0000 7fdab9d9b640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fda9c06be60 0x7fda9c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.885+0000 7fdab9d9b640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fda9c06be60 0x7fda9c06e320 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdab0034000 tx=0x7fdab002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:13.889+0000 7fdaaaffd640 1 -- 172.21.15.77:0/3224402153 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdabc1f7fa0 con 0x7fdabc0b9a90 2024-04-25T01:22:14.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.061+0000 7fdac0b5a640 1 -- 172.21.15.77:0/3224402153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdabc068fb0 con 0x7fdabc0b9a90 2024-04-25T01:22:14.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.062+0000 7fdaaaffd640 1 -- 172.21.15.77:0/3224402153 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7fdab4058790 con 0x7fdabc0b9a90 2024-04-25T01:22:14.043 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 -- 172.21.15.77:0/3224402153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fda9c06be60 msgr2=0x7fda9c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fda9c06be60 0x7fda9c06e320 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdab0034000 tx=0x7fdab002e000 comp rx=0 tx=0).stop 2024-04-25T01:22:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 -- 172.21.15.77:0/3224402153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdabc0b9a90 msgr2=0x7fdabc0ca3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdabc0b9a90 0x7fdabc0ca3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdab400c890 tx=0x7fdab400cd50 comp rx=0 tx=0).stop 2024-04-25T01:22:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 -- 172.21.15.77:0/3224402153 shutdown_connections 2024-04-25T01:22:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fda9c06be60 0x7fda9c06e320 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdabc0d27b0 0x7fdabc06be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdabc0d23d0 0x7fdabc1f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 --2- 172.21.15.77:0/3224402153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdabc0b9a90 0x7fdabc0ca3c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 -- 172.21.15.77:0/3224402153 >> 172.21.15.77:0/3224402153 conn(0x7fdabc0d3e30 msgr2=0x7fdabc0b8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 -- 172.21.15.77:0/3224402153 shutdown_connections 2024-04-25T01:22:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.064+0000 7fdac0b5a640 1 -- 172.21.15.77:0/3224402153 wait complete. 2024-04-25T01:22:14.056 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:14.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.176+0000 7fd0e351b640 1 Processor -- start 2024-04-25T01:22:14.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.176+0000 7fd0e351b640 1 -- start start 2024-04-25T01:22:14.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e351b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0dc15c2b0 0x7fd0dc0d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e351b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd0dc0b0c50 0x7fd0dc0d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e351b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0dc0d2810 con 0x7fd0dc0cfa30 2024-04-25T01:22:14.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e351b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fd0dc0cd920 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e351b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd0dc0cd2d0 con 0x7fd0dc0b0c50 2024-04-25T01:22:14.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e12b7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0dc15c2b0 0x7fd0dc0d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e1ab8640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd0dc0cfa30 0x7fd0dc0ce2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56604/0 (socket says 172.21.15.77:56604) 2024-04-25T01:22:14.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e1ab8640 1 -- 172.21.15.77:0/2523541534 learned_addr learned my addr 172.21.15.77:0/2523541534 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:14.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e12b7640 1 -- 172.21.15.77:0/2523541534 >> v2:172.21.15.121:3300/0 conn(0x7fd0dc0b0c50 msgr2=0x7fd0dc0d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:14.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e0ab6640 1 --2- 172.21.15.77:0/2523541534 >> v2:172.21.15.121:3300/0 conn(0x7fd0dc0b0c50 0x7fd0dc0d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e12b7640 1 --2- 172.21.15.77:0/2523541534 >> v2:172.21.15.121:3300/0 conn(0x7fd0dc0b0c50 0x7fd0dc0d0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e12b7640 1 -- 172.21.15.77:0/2523541534 >> v1:172.21.15.77:6789/0 conn(0x7fd0dc0cfa30 legacy=0x7fd0dc0ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0d3fff640 1 -- 172.21.15.77:0/2523541534 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2266670167 0 0) 0x7fd0dc0d2810 con 0x7fd0dc0cfa30 2024-04-25T01:22:14.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.177+0000 7fd0e12b7640 1 -- 172.21.15.77:0/2523541534 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0dc0cd5b0 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.178+0000 7fd0e12b7640 1 --2- 172.21.15.77:0/2523541534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc15c2b0 0x7fd0dc0d0230 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0cf750 tx=0x7fd0d802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f87305e8b0bba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:14.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.178+0000 7fd0d3fff640 1 -- 172.21.15.77:0/2523541534 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8037070 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.178+0000 7fd0d3fff640 1 -- 172.21.15.77:0/2523541534 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d8032440 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.178+0000 7fd0d3fff640 1 -- 172.21.15.77:0/2523541534 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8032810 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.179+0000 7fd0e351b640 1 -- 172.21.15.77:0/2523541534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc15c2b0 msgr2=0x7fd0dc0d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.179+0000 7fd0e351b640 1 --2- 172.21.15.77:0/2523541534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc15c2b0 0x7fd0dc0d0230 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0cf750 tx=0x7fd0d802a1a0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.179+0000 7fd0e351b640 1 -- 172.21.15.77:0/2523541534 shutdown_connections 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.179+0000 7fd0e351b640 1 --2- 172.21.15.77:0/2523541534 >> v2:172.21.15.121:3300/0 conn(0x7fd0dc0b0c50 0x7fd0dc0d0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.179+0000 7fd0e351b640 1 --2- 172.21.15.77:0/2523541534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc15c2b0 0x7fd0dc0d0230 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.179+0000 7fd0e351b640 1 -- 172.21.15.77:0/2523541534 >> 172.21.15.77:0/2523541534 conn(0x7fd0dc0d1c50 msgr2=0x7fd0dc0d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.179+0000 7fd0e351b640 1 -- 172.21.15.77:0/2523541534 shutdown_connections 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.179+0000 7fd0e351b640 1 -- 172.21.15.77:0/2523541534 wait complete. 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.180+0000 7fd0e351b640 1 Processor -- start 2024-04-25T01:22:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.180+0000 7fd0e351b640 1 -- start start 2024-04-25T01:22:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e351b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0dc0b0c50 0x7fd0dc0b64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e351b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0dc0cfa30 0x7fd0dc0cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e351b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc15c2b0 0x7fd0dc0aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e351b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc0b14d0 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e351b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc0b1350 con 0x7fd0dc0b0c50 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e351b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd0dc06ff50 con 0x7fd0dc0cfa30 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e1ab8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc15c2b0 0x7fd0dc0aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e1ab8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc15c2b0 0x7fd0dc0aff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56640/0 (socket says 172.21.15.77:56640) 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e1ab8640 1 -- 172.21.15.77:0/3347840794 learned_addr learned my addr 172.21.15.77:0/3347840794 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e12b7640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0dc0b0c50 0x7fd0dc0b64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e0ab6640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0dc0cfa30 0x7fd0dc0cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e1ab8640 1 -- 172.21.15.77:0/3347840794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0dc0cfa30 msgr2=0x7fd0dc0cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e1ab8640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0dc0cfa30 0x7fd0dc0cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e1ab8640 1 -- 172.21.15.77:0/3347840794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0dc0b0c50 msgr2=0x7fd0dc0b64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e1ab8640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0dc0b0c50 0x7fd0dc0b64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e1ab8640 1 -- 172.21.15.77:0/3347840794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0dc0c9020 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e0ab6640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0dc0cfa30 0x7fd0dc0cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:14.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e1ab8640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc15c2b0 0x7fd0dc0aff30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400c7d0 tx=0x7fd0d400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:14.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0d1ffb640 1 -- 172.21.15.77:0/3347840794 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4018070 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0d1ffb640 1 -- 172.21.15.77:0/3347840794 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d400a640 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0e351b640 1 -- 172.21.15.77:0/3347840794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0dc0b6a10 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.181+0000 7fd0d1ffb640 1 -- 172.21.15.77:0/3347840794 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d400aa70 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.182+0000 7fd0e351b640 1 -- 172.21.15.77:0/3347840794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0dc0af450 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.183+0000 7fd0e351b640 1 -- 172.21.15.77:0/3347840794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0dc068fb0 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.183+0000 7fd0d1ffb640 1 -- 172.21.15.77:0/3347840794 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd0d401b070 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.183+0000 7fd0d1ffb640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd0c006be30 0x7fd0c006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.184+0000 7fd0d1ffb640 1 -- 172.21.15.77:0/3347840794 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd0d408a720 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.184+0000 7fd0e12b7640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd0c006be30 0x7fd0c006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.184+0000 7fd0e12b7640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd0c006be30 0x7fd0c006e2f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8034000 tx=0x7fd0d802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:14.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.189+0000 7fd0d1ffb640 1 -- 172.21.15.77:0/3347840794 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd0d4058e20 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.361+0000 7fd0e351b640 1 -- 172.21.15.77:0/3347840794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd0dc0c5200 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.361+0000 7fd0d1ffb640 1 -- 172.21.15.77:0/3347840794 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7fd0d4058720 con 0x7fd0dc15c2b0 2024-04-25T01:22:14.342 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 -- 172.21.15.77:0/3347840794 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd0c006be30 msgr2=0x7fd0c006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd0c006be30 0x7fd0c006e2f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8034000 tx=0x7fd0d802e000 comp rx=0 tx=0).stop 2024-04-25T01:22:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 -- 172.21.15.77:0/3347840794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc15c2b0 msgr2=0x7fd0dc0aff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc15c2b0 0x7fd0dc0aff30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400c7d0 tx=0x7fd0d400cc90 comp rx=0 tx=0).stop 2024-04-25T01:22:14.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 -- 172.21.15.77:0/3347840794 shutdown_connections 2024-04-25T01:22:14.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd0c006be30 0x7fd0c006e2f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc15c2b0 0x7fd0dc0aff30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0dc0cfa30 0x7fd0dc0cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 --2- 172.21.15.77:0/3347840794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0dc0b0c50 0x7fd0dc0b64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 -- 172.21.15.77:0/3347840794 >> 172.21.15.77:0/3347840794 conn(0x7fd0dc0d1c50 msgr2=0x7fd0dc15e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:14.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 -- 172.21.15.77:0/3347840794 shutdown_connections 2024-04-25T01:22:14.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.363+0000 7fd0e351b640 1 -- 172.21.15.77:0/3347840794 wait complete. 2024-04-25T01:22:14.355 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:14.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.474+0000 7f72bfd89640 1 Processor -- start 2024-04-25T01:22:14.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.474+0000 7f72bfd89640 1 -- start start 2024-04-25T01:22:14.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.475+0000 7f72bfd89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b815e460 0x7f72b80d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.475+0000 7f72bfd89640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72b80cebe0 0x7f72b80ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.475+0000 7f72bfd89640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b80d0de0 con 0x7f72b80d09a0 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.475+0000 7f72bfd89640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f72b806c230 con 0x7f72b80cebe0 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.475+0000 7f72bfd89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72b80cca90 con 0x7f72b815e460 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.475+0000 7f72be326640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72b80cebe0 0x7f72b80ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.475+0000 7f72be326640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72b80cebe0 0x7f72b80ccf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56642/0 (socket says 172.21.15.77:56642) 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.475+0000 7f72be326640 1 -- 172.21.15.77:0/3340530386 learned_addr learned my addr 172.21.15.77:0/3340530386 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.476+0000 7f72bdb25640 1 --2- 172.21.15.77:0/3340530386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b815e460 0x7f72b80d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.476+0000 7f72be326640 1 -- 172.21.15.77:0/3340530386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b815e460 msgr2=0x7f72b80d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.476+0000 7f72be326640 1 --2- 172.21.15.77:0/3340530386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b815e460 0x7f72b80d2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.476+0000 7f72be326640 1 -- 172.21.15.77:0/3340530386 >> v1:172.21.15.121:6789/0 conn(0x7f72b80d09a0 legacy=0x7f72b80ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.476+0000 7f72be326640 1 -- 172.21.15.77:0/3340530386 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b8070280 con 0x7f72b80cebe0 2024-04-25T01:22:14.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.476+0000 7f72bdb25640 1 --2- 172.21.15.77:0/3340530386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b815e460 0x7f72b80d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:14.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.476+0000 7f72be326640 1 --2- 172.21.15.77:0/3340530386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72b80cebe0 0x7f72b80ccf20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009870 tx=0x7f72b002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebc678eeb74cf93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:14.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.476+0000 7f72bcb23640 1 -- 172.21.15.77:0/3340530386 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b0037070 con 0x7f72b80cebe0 2024-04-25T01:22:14.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.476+0000 7f72bcb23640 1 -- 172.21.15.77:0/3340530386 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72b0032440 con 0x7f72b80cebe0 2024-04-25T01:22:14.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.476+0000 7f72bcb23640 1 -- 172.21.15.77:0/3340530386 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b0032810 con 0x7f72b80cebe0 2024-04-25T01:22:14.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.477+0000 7f72bfd89640 1 -- 172.21.15.77:0/3340530386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72b80cebe0 msgr2=0x7f72b80ccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.477+0000 7f72bfd89640 1 --2- 172.21.15.77:0/3340530386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72b80cebe0 0x7f72b80ccf20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009870 tx=0x7f72b002a2a0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.477+0000 7f72bfd89640 1 -- 172.21.15.77:0/3340530386 shutdown_connections 2024-04-25T01:22:14.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.477+0000 7f72bfd89640 1 --2- 172.21.15.77:0/3340530386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72b80cebe0 0x7f72b80ccf20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.477+0000 7f72bfd89640 1 --2- 172.21.15.77:0/3340530386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b815e460 0x7f72b80d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.477+0000 7f72bfd89640 1 -- 172.21.15.77:0/3340530386 >> 172.21.15.77:0/3340530386 conn(0x7f72b80d1c50 msgr2=0x7f72b80d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:14.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.477+0000 7f72bfd89640 1 -- 172.21.15.77:0/3340530386 shutdown_connections 2024-04-25T01:22:14.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.477+0000 7f72bfd89640 1 -- 172.21.15.77:0/3340530386 wait complete. 2024-04-25T01:22:14.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.478+0000 7f72bfd89640 1 Processor -- start 2024-04-25T01:22:14.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.478+0000 7f72bfd89640 1 -- start start 2024-04-25T01:22:14.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bfd89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b80cebe0 0x7f72b81513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bfd89640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72b80d09a0 0x7f72b8150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bfd89640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b815e460 0x7f72b80c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bfd89640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72b80cc720 con 0x7f72b80d09a0 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bfd89640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f72b80cc440 con 0x7f72b815e460 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bfd89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72b80cd460 con 0x7f72b80cebe0 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bd324640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72b80d09a0 0x7f72b8150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bd324640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72b80d09a0 0x7f72b8150d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56654/0 (socket says 172.21.15.77:56654) 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bd324640 1 -- 172.21.15.77:0/3325517693 learned_addr learned my addr 172.21.15.77:0/3325517693 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72be326640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b815e460 0x7f72b80c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bdb25640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b80cebe0 0x7f72b81513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bd324640 1 -- 172.21.15.77:0/3325517693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b80cebe0 msgr2=0x7f72b81513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bd324640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b80cebe0 0x7f72b81513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bd324640 1 -- 172.21.15.77:0/3325517693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b815e460 msgr2=0x7f72b80c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bd324640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b815e460 0x7f72b80c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bd324640 1 -- 172.21.15.77:0/3325517693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b80c7c70 con 0x7f72b80d09a0 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72be326640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b815e460 0x7f72b80c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.479+0000 7f72bd324640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72b80d09a0 0x7f72b8150d20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f72a800c460 tx=0x7f72a800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:14.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.480+0000 7f72ae7fc640 1 -- 172.21.15.77:0/3325517693 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8016020 con 0x7f72b80d09a0 2024-04-25T01:22:14.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.480+0000 7f72ae7fc640 1 -- 172.21.15.77:0/3325517693 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72a8012870 con 0x7f72b80d09a0 2024-04-25T01:22:14.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.480+0000 7f72bfd89640 1 -- 172.21.15.77:0/3325517693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b815e840 con 0x7f72b80d09a0 2024-04-25T01:22:14.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.480+0000 7f72ae7fc640 1 -- 172.21.15.77:0/3325517693 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8012c60 con 0x7f72b80d09a0 2024-04-25T01:22:14.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.480+0000 7f72bfd89640 1 -- 172.21.15.77:0/3325517693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b81f79a0 con 0x7f72b80d09a0 2024-04-25T01:22:14.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.481+0000 7f72bfd89640 1 -- 172.21.15.77:0/3325517693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72b8068fb0 con 0x7f72b80d09a0 2024-04-25T01:22:14.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.481+0000 7f72ae7fc640 1 -- 172.21.15.77:0/3325517693 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f72a8019020 con 0x7f72b80d09a0 2024-04-25T01:22:14.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.481+0000 7f72ae7fc640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729c06be30 0x7f729c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.482+0000 7f72ae7fc640 1 -- 172.21.15.77:0/3325517693 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f72a808af20 con 0x7f72b80d09a0 2024-04-25T01:22:14.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.482+0000 7f72bdb25640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729c06be30 0x7f729c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.482+0000 7f72bdb25640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729c06be30 0x7f729c06e2f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f72b80b1580 tx=0x7f72b4008450 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:14.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.486+0000 7f72ae7fc640 1 -- 172.21.15.77:0/3325517693 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f72a8020030 con 0x7f72b80d09a0 2024-04-25T01:22:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.659+0000 7f72bfd89640 1 -- 172.21.15.77:0/3325517693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f72b806c560 con 0x7f72b80d09a0 2024-04-25T01:22:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.660+0000 7f72ae7fc640 1 -- 172.21.15.77:0/3325517693 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f72b80c7c70 con 0x7f72b80d09a0 2024-04-25T01:22:14.641 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 -- 172.21.15.77:0/3325517693 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729c06be30 msgr2=0x7f729c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729c06be30 0x7f729c06e2f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f72b80b1580 tx=0x7f72b4008450 comp rx=0 tx=0).stop 2024-04-25T01:22:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 -- 172.21.15.77:0/3325517693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72b80d09a0 msgr2=0x7f72b8150d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72b80d09a0 0x7f72b8150d20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f72a800c460 tx=0x7f72a800c920 comp rx=0 tx=0).stop 2024-04-25T01:22:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 -- 172.21.15.77:0/3325517693 shutdown_connections 2024-04-25T01:22:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729c06be30 0x7f729c06e2f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72b80d09a0 0x7f72b8150d20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b815e460 0x7f72b80c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 --2- 172.21.15.77:0/3325517693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b80cebe0 0x7f72b81513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 -- 172.21.15.77:0/3325517693 >> 172.21.15.77:0/3325517693 conn(0x7f72b80d1c50 msgr2=0x7f72b815bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 -- 172.21.15.77:0/3325517693 shutdown_connections 2024-04-25T01:22:14.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.662+0000 7f72bfd89640 1 -- 172.21.15.77:0/3325517693 wait complete. 2024-04-25T01:22:14.654 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:14.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.770+0000 7ffa895a9640 1 Processor -- start 2024-04-25T01:22:14.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.770+0000 7ffa895a9640 1 -- start start 2024-04-25T01:22:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa895a9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa8415e4c0 0x7ffa840d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa895a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa840d4a10 0x7ffa840b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa895a9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa840d1e90 con 0x7ffa840d2d40 2024-04-25T01:22:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa895a9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7ffa840b54a0 con 0x7ffa8415e4c0 2024-04-25T01:22:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa895a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffa840d2130 con 0x7ffa840d4a10 2024-04-25T01:22:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa82ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa8415e4c0 0x7ffa840d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa82ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa8415e4c0 0x7ffa840d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56662/0 (socket says 172.21.15.77:56662) 2024-04-25T01:22:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa82ffd640 1 -- 172.21.15.77:0/228716292 learned_addr learned my addr 172.21.15.77:0/228716292 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa7bfff640 1 --2- 172.21.15.77:0/228716292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa840d4a10 0x7ffa840b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa82ffd640 1 -- 172.21.15.77:0/228716292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa840d4a10 msgr2=0x7ffa840b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa82ffd640 1 --2- 172.21.15.77:0/228716292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa840d4a10 0x7ffa840b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa82ffd640 1 -- 172.21.15.77:0/228716292 >> v1:172.21.15.121:6789/0 conn(0x7ffa840d2d40 legacy=0x7ffa840d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.771+0000 7ffa82ffd640 1 -- 172.21.15.77:0/228716292 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa840b5780 con 0x7ffa8415e4c0 2024-04-25T01:22:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.772+0000 7ffa82ffd640 1 --2- 172.21.15.77:0/228716292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa8415e4c0 0x7ffa840d2800 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009870 tx=0x7ffa7402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28586ff6eadc6f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.772+0000 7ffa827fc640 1 -- 172.21.15.77:0/228716292 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74037070 con 0x7ffa8415e4c0 2024-04-25T01:22:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.772+0000 7ffa827fc640 1 -- 172.21.15.77:0/228716292 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa74032440 con 0x7ffa8415e4c0 2024-04-25T01:22:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.772+0000 7ffa827fc640 1 -- 172.21.15.77:0/228716292 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74032810 con 0x7ffa8415e4c0 2024-04-25T01:22:14.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.773+0000 7ffa895a9640 1 -- 172.21.15.77:0/228716292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa8415e4c0 msgr2=0x7ffa840d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.773+0000 7ffa895a9640 1 --2- 172.21.15.77:0/228716292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa8415e4c0 0x7ffa840d2800 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009870 tx=0x7ffa7402a1c0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.773+0000 7ffa895a9640 1 -- 172.21.15.77:0/228716292 shutdown_connections 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.773+0000 7ffa895a9640 1 --2- 172.21.15.77:0/228716292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa840d4a10 0x7ffa840b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.773+0000 7ffa895a9640 1 --2- 172.21.15.77:0/228716292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa8415e4c0 0x7ffa840d2800 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.773+0000 7ffa895a9640 1 -- 172.21.15.77:0/228716292 >> 172.21.15.77:0/228716292 conn(0x7ffa840d3ef0 msgr2=0x7ffa840d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.773+0000 7ffa895a9640 1 -- 172.21.15.77:0/228716292 shutdown_connections 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.773+0000 7ffa895a9640 1 -- 172.21.15.77:0/228716292 wait complete. 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.774+0000 7ffa895a9640 1 Processor -- start 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.774+0000 7ffa895a9640 1 -- start start 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa895a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa840d2d40 0x7ffa84151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa895a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa840d4a10 0x7ffa840b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa895a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa8415e4c0 0x7ffa840b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa895a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8415ea20 con 0x7ffa840d2d40 2024-04-25T01:22:14.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa895a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8415e8a0 con 0x7ffa840d4a10 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa895a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffa840d1010 con 0x7ffa8415e4c0 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa82ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa840d2d40 0x7ffa84151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa82ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa840d2d40 0x7ffa84151f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56672/0 (socket says 172.21.15.77:56672) 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa82ffd640 1 -- 172.21.15.77:0/2140402669 learned_addr learned my addr 172.21.15.77:0/2140402669 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa7bfff640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa840d4a10 0x7ffa840b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa837fe640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa8415e4c0 0x7ffa840b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa82ffd640 1 -- 172.21.15.77:0/2140402669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa8415e4c0 msgr2=0x7ffa840b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa82ffd640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa8415e4c0 0x7ffa840b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa82ffd640 1 -- 172.21.15.77:0/2140402669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa840d4a10 msgr2=0x7ffa840b0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa82ffd640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa840d4a10 0x7ffa840b0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa82ffd640 1 -- 172.21.15.77:0/2140402669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa840b1520 con 0x7ffa840d2d40 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa82ffd640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa840d2d40 0x7ffa84151f20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffa74000c00 tx=0x7ffa740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa7b7fe640 1 -- 172.21.15.77:0/2140402669 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74041070 con 0x7ffa840d2d40 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa7b7fe640 1 -- 172.21.15.77:0/2140402669 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa7403c890 con 0x7ffa840d2d40 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa895a9640 1 -- 172.21.15.77:0/2140402669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa840b1800 con 0x7ffa840d2d40 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa7b7fe640 1 -- 172.21.15.77:0/2140402669 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74037040 con 0x7ffa840d2d40 2024-04-25T01:22:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.775+0000 7ffa895a9640 1 -- 172.21.15.77:0/2140402669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa841f4ae0 con 0x7ffa840d2d40 2024-04-25T01:22:14.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.776+0000 7ffa895a9640 1 -- 172.21.15.77:0/2140402669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa84068fb0 con 0x7ffa840d2d40 2024-04-25T01:22:14.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.777+0000 7ffa7b7fe640 1 -- 172.21.15.77:0/2140402669 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ffa74044070 con 0x7ffa840d2d40 2024-04-25T01:22:14.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.777+0000 7ffa7b7fe640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa6406be30 0x7ffa6406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:14.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.777+0000 7ffa7b7fe640 1 -- 172.21.15.77:0/2140402669 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ffa740abd30 con 0x7ffa840d2d40 2024-04-25T01:22:14.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.777+0000 7ffa7bfff640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa6406be30 0x7ffa6406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:14.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.778+0000 7ffa7bfff640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa6406be30 0x7ffa6406e2f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa840b1b00 tx=0x7ffa6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.781+0000 7ffa7b7fe640 1 -- 172.21.15.77:0/2140402669 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffa7407a4c0 con 0x7ffa840d2d40 2024-04-25T01:22:14.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.990+0000 7ffa895a9640 1 -- 172.21.15.77:0/2140402669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa840cad10 con 0x7ffa840d2d40 2024-04-25T01:22:14.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.991+0000 7ffa7b7fe640 1 -- 172.21.15.77:0/2140402669 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7ffa74079dc0 con 0x7ffa840d2d40 2024-04-25T01:22:14.972 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:14.972 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4267907621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4267907621},{"type":"v1","addr":"172.21.15.77:6837","nonce":4267907621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/493518847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":493518847},{"type":"v1","addr":"172.21.15.121:6833","nonce":493518847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:21:45.408154+0000","modified":"2024-04-25T01:22:06.554124+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6835/2810316639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2810316639},{"type":"v1","addr":"172.21.15.121:6835","nonce":2810316639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T01:22:14.972 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2024-04-25T01:22:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 -- 172.21.15.77:0/2140402669 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa6406be30 msgr2=0x7ffa6406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa6406be30 0x7ffa6406e2f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa840b1b00 tx=0x7ffa6c008040 comp rx=0 tx=0).stop 2024-04-25T01:22:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 -- 172.21.15.77:0/2140402669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa840d2d40 msgr2=0x7ffa84151f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa840d2d40 0x7ffa84151f20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffa74000c00 tx=0x7ffa740046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 -- 172.21.15.77:0/2140402669 shutdown_connections 2024-04-25T01:22:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa6406be30 0x7ffa6406e2f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa8415e4c0 0x7ffa840b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa840d4a10 0x7ffa840b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 --2- 172.21.15.77:0/2140402669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa840d2d40 0x7ffa84151f20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 -- 172.21.15.77:0/2140402669 >> 172.21.15.77:0/2140402669 conn(0x7ffa840d3ef0 msgr2=0x7ffa840b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 -- 172.21.15.77:0/2140402669 shutdown_connections 2024-04-25T01:22:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:14.993+0000 7ffa895a9640 1 -- 172.21.15.77:0/2140402669 wait complete. 2024-04-25T01:22:14.985 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.100+0000 7ff322fe7640 1 Processor -- start 2024-04-25T01:22:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.100+0000 7ff322fe7640 1 -- start start 2024-04-25T01:22:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.101+0000 7ff322fe7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff31c154ac0 0x7ff31c15b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.101+0000 7ff322fe7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff31c154ea0 con 0x7ff31c06e550 2024-04-25T01:22:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.101+0000 7ff322fe7640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff31c06d790 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.101+0000 7ff322fe7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff31c06d540 con 0x7ff31c154ac0 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.101+0000 7ff321584640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff31c06e550 0x7ff31c06e960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56612/0 (socket says 172.21.15.77:56612) 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.101+0000 7ff321584640 1 -- 172.21.15.77:0/3776856008 learned_addr learned my addr 172.21.15.77:0/3776856008 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.102+0000 7ff320d83640 1 --2- 172.21.15.77:0/3776856008 >> v2:172.21.15.121:3300/0 conn(0x7ff31c154ac0 0x7ff31c15b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.102+0000 7ff31b7fe640 1 -- 172.21.15.77:0/3776856008 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1552107499 0 0) 0x7ff31c154ea0 con 0x7ff31c06e550 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.102+0000 7ff31b7fe640 1 -- 172.21.15.77:0/3776856008 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff304003520 con 0x7ff31c06e550 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.102+0000 7ff31b7fe640 1 -- 172.21.15.77:0/3776856008 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3188379947 0 0) 0x7ff31c06d790 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.102+0000 7ff31b7fe640 1 -- 172.21.15.77:0/3776856008 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff31c154ea0 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.102+0000 7ff31b7fe640 1 -- 172.21.15.77:0/3776856008 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1350236095 0 0) 0x7ff304003520 con 0x7ff31c06e550 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.102+0000 7ff320d83640 1 -- 172.21.15.77:0/3776856008 >> v1:172.21.15.121:6789/0 conn(0x7ff31c0cf2a0 legacy=0x7ff31c0711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.102+0000 7ff320d83640 1 -- 172.21.15.77:0/3776856008 >> v1:172.21.15.77:6789/0 conn(0x7ff31c06e550 legacy=0x7ff31c06e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.102+0000 7ff320d83640 1 -- 172.21.15.77:0/3776856008 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31c06da70 con 0x7ff31c154ac0 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.103+0000 7ff320d83640 1 --2- 172.21.15.77:0/3776856008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff31c154ac0 0x7ff31c15b5f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0097a0 tx=0x7ff30c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85153e9b572fbddb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.103+0000 7ff31b7fe640 1 -- 172.21.15.77:0/3776856008 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c037070 con 0x7ff31c154ac0 2024-04-25T01:22:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.103+0000 7ff31b7fe640 1 -- 172.21.15.77:0/3776856008 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff30c02ac50 con 0x7ff31c154ac0 2024-04-25T01:22:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.103+0000 7ff31b7fe640 1 -- 172.21.15.77:0/3776856008 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3140019f0 con 0x7ff31c154ac0 2024-04-25T01:22:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.103+0000 7ff322fe7640 1 -- 172.21.15.77:0/3776856008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff31c154ac0 msgr2=0x7ff31c15b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.103+0000 7ff322fe7640 1 --2- 172.21.15.77:0/3776856008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff31c154ac0 0x7ff31c15b5f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0097a0 tx=0x7ff30c02a1c0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.103+0000 7ff322fe7640 1 -- 172.21.15.77:0/3776856008 shutdown_connections 2024-04-25T01:22:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.103+0000 7ff322fe7640 1 --2- 172.21.15.77:0/3776856008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff31c154ac0 0x7ff31c15b5f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.103+0000 7ff322fe7640 1 -- 172.21.15.77:0/3776856008 >> 172.21.15.77:0/3776856008 conn(0x7ff31c0b1820 msgr2=0x7ff31c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.103+0000 7ff322fe7640 1 -- 172.21.15.77:0/3776856008 shutdown_connections 2024-04-25T01:22:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.103+0000 7ff322fe7640 1 -- 172.21.15.77:0/3776856008 wait complete. 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.104+0000 7ff322fe7640 1 Processor -- start 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.105+0000 7ff322fe7640 1 -- start start 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.105+0000 7ff322fe7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31c06e550 0x7ff31c14a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.105+0000 7ff322fe7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff31c0cf2a0 0x7ff31c1324a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.105+0000 7ff322fe7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff31c154ac0 0x7ff31c148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.105+0000 7ff322fe7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c071900 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.105+0000 7ff322fe7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c06e030 con 0x7ff31c154ac0 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.105+0000 7ff322fe7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff31c0cf750 con 0x7ff31c06e550 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.105+0000 7ff31bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff31c0cf2a0 0x7ff31c1324a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff31bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff31c0cf2a0 0x7ff31c1324a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56682/0 (socket says 172.21.15.77:56682) 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff31bfff640 1 -- 172.21.15.77:0/3836835387 learned_addr learned my addr 172.21.15.77:0/3836835387 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff320d83640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31c06e550 0x7ff31c14a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff321584640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff31c154ac0 0x7ff31c148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff31bfff640 1 -- 172.21.15.77:0/3836835387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31c06e550 msgr2=0x7ff31c14a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff31bfff640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31c06e550 0x7ff31c14a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff31bfff640 1 -- 172.21.15.77:0/3836835387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff31c154ac0 msgr2=0x7ff31c148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff31bfff640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff31c154ac0 0x7ff31c148410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff31bfff640 1 -- 172.21.15.77:0/3836835387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31c132ba0 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff31bfff640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff31c0cf2a0 0x7ff31c1324a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff31000c490 tx=0x7ff31000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff3197fa640 1 -- 172.21.15.77:0/3836835387 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3100052d0 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff3197fa640 1 -- 172.21.15.77:0/3836835387 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff310005470 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff322fe7640 1 -- 172.21.15.77:0/3836835387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff31c132e80 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff3197fa640 1 -- 172.21.15.77:0/3836835387 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff310012850 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.106+0000 7ff322fe7640 1 -- 172.21.15.77:0/3836835387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff31c133440 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.107+0000 7ff322fe7640 1 -- 172.21.15.77:0/3836835387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff31c068fb0 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.108+0000 7ff3197fa640 1 -- 172.21.15.77:0/3836835387 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff3100129f0 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.108+0000 7ff3197fa640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff30806be60 0x7ff30806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.108+0000 7ff320d83640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff30806be60 0x7ff30806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.108+0000 7ff3197fa640 1 -- 172.21.15.77:0/3836835387 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff31008ad00 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.109+0000 7ff320d83640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff30806be60 0x7ff30806e320 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff30c034000 tx=0x7ff30c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:15.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.112+0000 7ff3197fa640 1 -- 172.21.15.77:0/3836835387 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff310058d20 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.289+0000 7ff322fe7640 1 -- 172.21.15.77:0/3836835387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff31c169250 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.290+0000 7ff3197fa640 1 -- 172.21.15.77:0/3836835387 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7ff31005f020 con 0x7ff31c0cf2a0 2024-04-25T01:22:15.271 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:15.272 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:15.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.292+0000 7ff322fe7640 1 -- 172.21.15.77:0/3836835387 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff30806be60 msgr2=0x7ff30806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.292+0000 7ff322fe7640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff30806be60 0x7ff30806e320 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff30c034000 tx=0x7ff30c02e000 comp rx=0 tx=0).stop 2024-04-25T01:22:15.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.292+0000 7ff322fe7640 1 -- 172.21.15.77:0/3836835387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff31c0cf2a0 msgr2=0x7ff31c1324a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.292+0000 7ff322fe7640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff31c0cf2a0 0x7ff31c1324a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff31000c490 tx=0x7ff31000c950 comp rx=0 tx=0).stop 2024-04-25T01:22:15.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.293+0000 7ff322fe7640 1 -- 172.21.15.77:0/3836835387 shutdown_connections 2024-04-25T01:22:15.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.293+0000 7ff322fe7640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff30806be60 0x7ff30806e320 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.293+0000 7ff322fe7640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff31c0cf2a0 0x7ff31c1324a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.293+0000 7ff322fe7640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff31c154ac0 0x7ff31c148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.293+0000 7ff322fe7640 1 --2- 172.21.15.77:0/3836835387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31c06e550 0x7ff31c14a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.293+0000 7ff322fe7640 1 -- 172.21.15.77:0/3836835387 >> 172.21.15.77:0/3836835387 conn(0x7ff31c0b1820 msgr2=0x7ff31c074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:15.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.293+0000 7ff322fe7640 1 -- 172.21.15.77:0/3836835387 shutdown_connections 2024-04-25T01:22:15.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.293+0000 7ff322fe7640 1 -- 172.21.15.77:0/3836835387 wait complete. 2024-04-25T01:22:15.285 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:15.285 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:15.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.403+0000 7f6d0b695640 1 Processor -- start 2024-04-25T01:22:15.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.404+0000 7f6d0b695640 1 -- start start 2024-04-25T01:22:15.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.404+0000 7f6d0b695640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d04158a70 0x7f6d040b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.404+0000 7f6d0b695640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d0415bc30 0x7f6d040b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.404+0000 7f6d0b695640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0415c210 con 0x7f6d040b37c0 2024-04-25T01:22:15.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.404+0000 7f6d0b695640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6d040b2ce0 con 0x7f6d0415bc30 2024-04-25T01:22:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.404+0000 7f6d0b695640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6d040b2fc0 con 0x7f6d04158a70 2024-04-25T01:22:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.405+0000 7f6d09c32640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d0415bc30 0x7f6d040b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.405+0000 7f6d09c32640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d0415bc30 0x7f6d040b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56694/0 (socket says 172.21.15.77:56694) 2024-04-25T01:22:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.405+0000 7f6d09c32640 1 -- 172.21.15.77:0/2707606607 learned_addr learned my addr 172.21.15.77:0/2707606607 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.405+0000 7f6d09431640 1 --2- 172.21.15.77:0/2707606607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d04158a70 0x7f6d040b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.405+0000 7f6d09c32640 1 -- 172.21.15.77:0/2707606607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d04158a70 msgr2=0x7f6d040b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.405+0000 7f6d09c32640 1 --2- 172.21.15.77:0/2707606607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d04158a70 0x7f6d040b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.405+0000 7f6d09c32640 1 -- 172.21.15.77:0/2707606607 >> v1:172.21.15.121:6789/0 conn(0x7f6d040b37c0 legacy=0x7f6d0415b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.405+0000 7f6d09c32640 1 -- 172.21.15.77:0/2707606607 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d040b32a0 con 0x7f6d0415bc30 2024-04-25T01:22:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6d09c32640 1 --2- 172.21.15.77:0/2707606607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d0415bc30 0x7f6d040b4600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009e50 tx=0x7f6cfc02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0492505e31a8e58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:15.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6cf7fff640 1 -- 172.21.15.77:0/2707606607 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc036070 con 0x7f6d0415bc30 2024-04-25T01:22:15.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6cf7fff640 1 -- 172.21.15.77:0/2707606607 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cfc031600 con 0x7f6d0415bc30 2024-04-25T01:22:15.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6cf7fff640 1 -- 172.21.15.77:0/2707606607 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc0319d0 con 0x7f6d0415bc30 2024-04-25T01:22:15.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6d0b695640 1 -- 172.21.15.77:0/2707606607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d0415bc30 msgr2=0x7f6d040b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6d0b695640 1 --2- 172.21.15.77:0/2707606607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d0415bc30 0x7f6d040b4600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009e50 tx=0x7f6cfc02a710 comp rx=0 tx=0).stop 2024-04-25T01:22:15.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6d0b695640 1 -- 172.21.15.77:0/2707606607 shutdown_connections 2024-04-25T01:22:15.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6d0b695640 1 --2- 172.21.15.77:0/2707606607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d0415bc30 0x7f6d040b4600 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6d0b695640 1 --2- 172.21.15.77:0/2707606607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d04158a70 0x7f6d040b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6d0b695640 1 -- 172.21.15.77:0/2707606607 >> 172.21.15.77:0/2707606607 conn(0x7f6d0406b720 msgr2=0x7f6d040b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:15.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6d0b695640 1 -- 172.21.15.77:0/2707606607 shutdown_connections 2024-04-25T01:22:15.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.406+0000 7f6d0b695640 1 -- 172.21.15.77:0/2707606607 wait complete. 2024-04-25T01:22:15.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.407+0000 7f6d0b695640 1 Processor -- start 2024-04-25T01:22:15.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d0b695640 1 -- start start 2024-04-25T01:22:15.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d0b695640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d040b37c0 0x7f6d040abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d0b695640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d04158a70 0x7f6d040ae650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d0b695640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d0415bc30 0x7f6d040aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d0b695640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0406cb70 con 0x7f6d0415bc30 2024-04-25T01:22:15.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d0b695640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0406c9f0 con 0x7f6d040b37c0 2024-04-25T01:22:15.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d0b695640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0406ccf0 con 0x7f6d04158a70 2024-04-25T01:22:15.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d09c32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d0415bc30 0x7f6d040aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d09c32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d0415bc30 0x7f6d040aeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56704/0 (socket says 172.21.15.77:56704) 2024-04-25T01:22:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d09c32640 1 -- 172.21.15.77:0/2418074709 learned_addr learned my addr 172.21.15.77:0/2418074709 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d09431640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d040b37c0 0x7f6d040abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.408+0000 7f6d08c30640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d04158a70 0x7f6d040ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.409+0000 7f6d09c32640 1 -- 172.21.15.77:0/2418074709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d04158a70 msgr2=0x7f6d040ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.409+0000 7f6d09c32640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d04158a70 0x7f6d040ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.409+0000 7f6d09c32640 1 -- 172.21.15.77:0/2418074709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d040b37c0 msgr2=0x7f6d040abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.409+0000 7f6d09c32640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d040b37c0 0x7f6d040abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.409+0000 7f6d09c32640 1 -- 172.21.15.77:0/2418074709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d04141590 con 0x7f6d0415bc30 2024-04-25T01:22:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.409+0000 7f6d09c32640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d0415bc30 0x7f6d040aeb90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009f70 tx=0x7f6cfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.409+0000 7f6cf5ffb640 1 -- 172.21.15.77:0/2418074709 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc03f020 con 0x7f6d0415bc30 2024-04-25T01:22:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.409+0000 7f6cf5ffb640 1 -- 172.21.15.77:0/2418074709 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cfc037040 con 0x7f6d0415bc30 2024-04-25T01:22:15.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.409+0000 7f6cf5ffb640 1 -- 172.21.15.77:0/2418074709 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc036070 con 0x7f6d0415bc30 2024-04-25T01:22:15.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.409+0000 7f6d0b695640 1 -- 172.21.15.77:0/2418074709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d040af230 con 0x7f6d0415bc30 2024-04-25T01:22:15.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.409+0000 7f6d0b695640 1 -- 172.21.15.77:0/2418074709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d040cd640 con 0x7f6d0415bc30 2024-04-25T01:22:15.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.410+0000 7f6d0b695640 1 -- 172.21.15.77:0/2418074709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d04068fb0 con 0x7f6d0415bc30 2024-04-25T01:22:15.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.411+0000 7f6cf5ffb640 1 -- 172.21.15.77:0/2418074709 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6cfc042020 con 0x7f6d0415bc30 2024-04-25T01:22:15.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.411+0000 7f6cf5ffb640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6ce806be30 0x7f6ce806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.411+0000 7f6cf5ffb640 1 -- 172.21.15.77:0/2418074709 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6cfc0ab520 con 0x7f6d0415bc30 2024-04-25T01:22:15.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.411+0000 7f6d09431640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6ce806be30 0x7f6ce806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.411+0000 7f6d09431640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6ce806be30 0x7f6ce806e2f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006fd0 tx=0x7f6cf8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:15.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.415+0000 7f6cf5ffb640 1 -- 172.21.15.77:0/2418074709 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6cfc049030 con 0x7f6d0415bc30 2024-04-25T01:22:15.481 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:22:15.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:22:15.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.589+0000 7f6d0b695640 1 -- 172.21.15.77:0/2418074709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6d040b3c70 con 0x7f6d0415bc30 2024-04-25T01:22:15.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.589+0000 7f6cf5ffb640 1 -- 172.21.15.77:0/2418074709 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f6d04141590 con 0x7f6d0415bc30 2024-04-25T01:22:15.570 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:15.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 -- 172.21.15.77:0/2418074709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6ce806be30 msgr2=0x7f6ce806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6ce806be30 0x7f6ce806e2f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006fd0 tx=0x7f6cf8008040 comp rx=0 tx=0).stop 2024-04-25T01:22:15.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 -- 172.21.15.77:0/2418074709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d0415bc30 msgr2=0x7f6d040aeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d0415bc30 0x7f6d040aeb90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009f70 tx=0x7f6cfc004a40 comp rx=0 tx=0).stop 2024-04-25T01:22:15.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 -- 172.21.15.77:0/2418074709 shutdown_connections 2024-04-25T01:22:15.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6ce806be30 0x7f6ce806e2f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d0415bc30 0x7f6d040aeb90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d04158a70 0x7f6d040ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 --2- 172.21.15.77:0/2418074709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d040b37c0 0x7f6d040abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 -- 172.21.15.77:0/2418074709 >> 172.21.15.77:0/2418074709 conn(0x7f6d0406b720 msgr2=0x7f6d040b46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:15.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 -- 172.21.15.77:0/2418074709 shutdown_connections 2024-04-25T01:22:15.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.591+0000 7f6d0b695640 1 -- 172.21.15.77:0/2418074709 wait complete. 2024-04-25T01:22:15.583 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:15.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.743+0000 7fed4b03f640 1 Processor -- start 2024-04-25T01:22:15.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.743+0000 7fed4b03f640 1 -- start start 2024-04-25T01:22:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed4b03f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed44071890 0x7fed44141680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed4b03f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed4406d540 0x7fed4406e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed4b03f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed4415e980 con 0x7fed44162590 2024-04-25T01:22:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed4b03f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fed4415ec60 con 0x7fed44071890 2024-04-25T01:22:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed4b03f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed4415ef40 con 0x7fed4406d540 2024-04-25T01:22:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed48ddb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed44071890 0x7fed44141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed48ddb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed44071890 0x7fed44141680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56720/0 (socket says 172.21.15.77:56720) 2024-04-25T01:22:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed48ddb640 1 -- 172.21.15.77:0/4052431839 learned_addr learned my addr 172.21.15.77:0/4052431839 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed495dc640 1 --2- 172.21.15.77:0/4052431839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed4406d540 0x7fed4406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed48ddb640 1 -- 172.21.15.77:0/4052431839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed4406d540 msgr2=0x7fed4406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed48ddb640 1 --2- 172.21.15.77:0/4052431839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed4406d540 0x7fed4406e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed48ddb640 1 -- 172.21.15.77:0/4052431839 >> v1:172.21.15.121:6789/0 conn(0x7fed44162590 legacy=0x7fed4406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed48ddb640 1 -- 172.21.15.77:0/4052431839 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4415f220 con 0x7fed44071890 2024-04-25T01:22:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.744+0000 7fed495dc640 1 --2- 172.21.15.77:0/4052431839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed4406d540 0x7fed4406e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.745+0000 7fed48ddb640 1 --2- 172.21.15.77:0/4052431839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed44071890 0x7fed44141680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fed38009870 tx=0x7fed3802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5e1bad455f3f6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.745+0000 7fed437fe640 1 -- 172.21.15.77:0/4052431839 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38037070 con 0x7fed44071890 2024-04-25T01:22:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.745+0000 7fed437fe640 1 -- 172.21.15.77:0/4052431839 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed38032440 con 0x7fed44071890 2024-04-25T01:22:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.745+0000 7fed437fe640 1 -- 172.21.15.77:0/4052431839 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38032810 con 0x7fed44071890 2024-04-25T01:22:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.745+0000 7fed4b03f640 1 -- 172.21.15.77:0/4052431839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed44071890 msgr2=0x7fed44141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.745+0000 7fed4b03f640 1 --2- 172.21.15.77:0/4052431839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed44071890 0x7fed44141680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fed38009870 tx=0x7fed3802a200 comp rx=0 tx=0).stop 2024-04-25T01:22:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.746+0000 7fed4b03f640 1 -- 172.21.15.77:0/4052431839 shutdown_connections 2024-04-25T01:22:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.746+0000 7fed4b03f640 1 --2- 172.21.15.77:0/4052431839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed4406d540 0x7fed4406e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.746+0000 7fed4b03f640 1 --2- 172.21.15.77:0/4052431839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed44071890 0x7fed44141680 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.746+0000 7fed4b03f640 1 -- 172.21.15.77:0/4052431839 >> 172.21.15.77:0/4052431839 conn(0x7fed440b1c60 msgr2=0x7fed440b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.746+0000 7fed4b03f640 1 -- 172.21.15.77:0/4052431839 shutdown_connections 2024-04-25T01:22:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.746+0000 7fed4b03f640 1 -- 172.21.15.77:0/4052431839 wait complete. 2024-04-25T01:22:15.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.747+0000 7fed4b03f640 1 Processor -- start 2024-04-25T01:22:15.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.747+0000 7fed4b03f640 1 -- start start 2024-04-25T01:22:15.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed4b03f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed4406d540 0x7fed441f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed4b03f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed44071890 0x7fed4413e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed4b03f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed44162590 0x7fed4413ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed4b03f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fed4415e590 con 0x7fed44071890 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed4b03f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fed44162a40 con 0x7fed4406d540 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed4b03f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed4415e710 con 0x7fed44162590 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed43fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed44071890 0x7fed4413e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed43fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed44071890 0x7fed4413e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56732/0 (socket says 172.21.15.77:56732) 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed43fff640 1 -- 172.21.15.77:0/583985993 learned_addr learned my addr 172.21.15.77:0/583985993 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed48ddb640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed4406d540 0x7fed441f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed495dc640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed44162590 0x7fed4413ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed43fff640 1 -- 172.21.15.77:0/583985993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed44162590 msgr2=0x7fed4413ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed43fff640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed44162590 0x7fed4413ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed43fff640 1 -- 172.21.15.77:0/583985993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed4406d540 msgr2=0x7fed441f3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed43fff640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed4406d540 0x7fed441f3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed43fff640 1 -- 172.21.15.77:0/583985993 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed441f7a50 con 0x7fed44071890 2024-04-25T01:22:15.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed43fff640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed44071890 0x7fed4413e840 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fed3000b420 tx=0x7fed3000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:15.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed417fa640 1 -- 172.21.15.77:0/583985993 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed30011830 con 0x7fed44071890 2024-04-25T01:22:15.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed417fa640 1 -- 172.21.15.77:0/583985993 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed30016070 con 0x7fed44071890 2024-04-25T01:22:15.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed417fa640 1 -- 172.21.15.77:0/583985993 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed30011c10 con 0x7fed44071890 2024-04-25T01:22:15.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed4b03f640 1 -- 172.21.15.77:0/583985993 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed441f7d30 con 0x7fed44071890 2024-04-25T01:22:15.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.748+0000 7fed4b03f640 1 -- 172.21.15.77:0/583985993 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4415df50 con 0x7fed44071890 2024-04-25T01:22:15.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.749+0000 7fed4b03f640 1 -- 172.21.15.77:0/583985993 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed44068fb0 con 0x7fed44071890 2024-04-25T01:22:15.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.750+0000 7fed417fa640 1 -- 172.21.15.77:0/583985993 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fed3001b460 con 0x7fed44071890 2024-04-25T01:22:15.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.750+0000 7fed417fa640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed2806be30 0x7fed2806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:15.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.750+0000 7fed417fa640 1 -- 172.21.15.77:0/583985993 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fed3008aaf0 con 0x7fed44071890 2024-04-25T01:22:15.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.750+0000 7fed48ddb640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed2806be30 0x7fed2806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:15.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.751+0000 7fed48ddb640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed2806be30 0x7fed2806e2f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fed38034000 tx=0x7fed3802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:15.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.754+0000 7fed417fa640 1 -- 172.21.15.77:0/583985993 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fed30020020 con 0x7fed44071890 2024-04-25T01:22:15.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.928+0000 7fed4b03f640 1 -- 172.21.15.77:0/583985993 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fed4406f770 con 0x7fed44071890 2024-04-25T01:22:15.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.928+0000 7fed417fa640 1 -- 172.21.15.77:0/583985993 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7fed30058b10 con 0x7fed44071890 2024-04-25T01:22:15.909 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 -- 172.21.15.77:0/583985993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed2806be30 msgr2=0x7fed2806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed2806be30 0x7fed2806e2f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fed38034000 tx=0x7fed3802e000 comp rx=0 tx=0).stop 2024-04-25T01:22:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 -- 172.21.15.77:0/583985993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed44071890 msgr2=0x7fed4413e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed44071890 0x7fed4413e840 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fed3000b420 tx=0x7fed3000b8e0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 -- 172.21.15.77:0/583985993 shutdown_connections 2024-04-25T01:22:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed2806be30 0x7fed2806e2f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed44071890 0x7fed4413e840 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed44162590 0x7fed4413ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 --2- 172.21.15.77:0/583985993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed4406d540 0x7fed441f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 -- 172.21.15.77:0/583985993 >> 172.21.15.77:0/583985993 conn(0x7fed440b1c60 msgr2=0x7fed440b4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 -- 172.21.15.77:0/583985993 shutdown_connections 2024-04-25T01:22:15.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:15.930+0000 7fed4b03f640 1 -- 172.21.15.77:0/583985993 wait complete. 2024-04-25T01:22:15.921 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.040+0000 7f4b6cf4f640 1 Processor -- start 2024-04-25T01:22:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.040+0000 7f4b6cf4f640 1 -- start start 2024-04-25T01:22:16.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b6cf4f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4b68158a70 0x7f4b680b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b6cf4f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b680b37c0 0x7f4b680b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b6cf4f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6815c210 con 0x7f4b6815b520 2024-04-25T01:22:16.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b6cf4f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4b680b2ce0 con 0x7f4b68158a70 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b6cf4f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4b680b2fc0 con 0x7f4b680b37c0 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b6659c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4b68158a70 0x7f4b680b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b5ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b680b37c0 0x7f4b680b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b5ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b680b37c0 0x7f4b680b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:34216/0 (socket says 172.21.15.77:34216) 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b66d9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4b6815b520 0x7f4b6815b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:59758/0 (socket says 172.21.15.77:59758) 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b6659c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4b68158a70 0x7f4b680b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:49114/0 (socket says 172.21.15.77:49114) 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b5ffff640 1 -- 172.21.15.77:0/3543019694 learned_addr learned my addr 172.21.15.77:0/3543019694 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b5ffff640 1 -- 172.21.15.77:0/3543019694 >> v2:172.21.15.121:3300/0 conn(0x7f4b68158a70 msgr2=0x7f4b680b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b5ffff640 1 --2- 172.21.15.77:0/3543019694 >> v2:172.21.15.121:3300/0 conn(0x7f4b68158a70 0x7f4b680b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b5ffff640 1 -- 172.21.15.77:0/3543019694 >> v1:172.21.15.121:6789/0 conn(0x7f4b6815b520 legacy=0x7f4b6815b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b65d9b640 1 -- 172.21.15.77:0/3543019694 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1194275996 0 0) 0x7f4b6815c210 con 0x7f4b6815b520 2024-04-25T01:22:16.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.041+0000 7f4b5ffff640 1 -- 172.21.15.77:0/3543019694 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b680b32a0 con 0x7f4b680b37c0 2024-04-25T01:22:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.042+0000 7f4b5ffff640 1 --2- 172.21.15.77:0/3543019694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b680b37c0 0x7f4b680b4600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000bb10 tx=0x7f4b5002c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85586543c7612455 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.042+0000 7f4b65d9b640 1 -- 172.21.15.77:0/3543019694 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50036020 con 0x7f4b680b37c0 2024-04-25T01:22:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.042+0000 7f4b65d9b640 1 -- 172.21.15.77:0/3543019694 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b50032820 con 0x7f4b680b37c0 2024-04-25T01:22:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.042+0000 7f4b65d9b640 1 -- 172.21.15.77:0/3543019694 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50032bf0 con 0x7f4b680b37c0 2024-04-25T01:22:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.043+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/3543019694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b680b37c0 msgr2=0x7f4b680b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.043+0000 7f4b6cf4f640 1 --2- 172.21.15.77:0/3543019694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b680b37c0 0x7f4b680b4600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000bb10 tx=0x7f4b5002c920 comp rx=0 tx=0).stop 2024-04-25T01:22:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.043+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/3543019694 shutdown_connections 2024-04-25T01:22:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.043+0000 7f4b6cf4f640 1 --2- 172.21.15.77:0/3543019694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b680b37c0 0x7f4b680b4600 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.043+0000 7f4b6cf4f640 1 --2- 172.21.15.77:0/3543019694 >> v2:172.21.15.121:3300/0 conn(0x7f4b68158a70 0x7f4b680b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.043+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/3543019694 >> 172.21.15.77:0/3543019694 conn(0x7f4b6806b720 msgr2=0x7f4b680b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.043+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/3543019694 shutdown_connections 2024-04-25T01:22:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.043+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/3543019694 wait complete. 2024-04-25T01:22:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.044+0000 7f4b6cf4f640 1 Processor -- start 2024-04-25T01:22:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.044+0000 7f4b6cf4f640 1 -- start start 2024-04-25T01:22:16.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b6cf4f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b680b37c0 0x7f4b6815d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b6cf4f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b68158a70 0x7f4b681330e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b6cf4f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b6815b520 0x7f4b680b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b6cf4f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6806cb70 con 0x7f4b6815b520 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b6cf4f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6806c9f0 con 0x7f4b680b37c0 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b6cf4f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6806ccf0 con 0x7f4b68158a70 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b66d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b6815b520 0x7f4b680b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b66d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b6815b520 0x7f4b680b0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56734/0 (socket says 172.21.15.77:56734) 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b66d9d640 1 -- 172.21.15.77:0/1698609351 learned_addr learned my addr 172.21.15.77:0/1698609351 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b6659c640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b680b37c0 0x7f4b6815d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b5ffff640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b68158a70 0x7f4b681330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b66d9d640 1 -- 172.21.15.77:0/1698609351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b68158a70 msgr2=0x7f4b681330e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b66d9d640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b68158a70 0x7f4b681330e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b66d9d640 1 -- 172.21.15.77:0/1698609351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b680b37c0 msgr2=0x7f4b6815d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b66d9d640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b680b37c0 0x7f4b6815d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b66d9d640 1 -- 172.21.15.77:0/1698609351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b680b12d0 con 0x7f4b6815b520 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.045+0000 7f4b5ffff640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b68158a70 0x7f4b681330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.046+0000 7f4b66d9d640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b6815b520 0x7f4b680b0d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000c780 tx=0x7f4b6000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.046+0000 7f4b5effd640 1 -- 172.21.15.77:0/1698609351 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60018070 con 0x7f4b6815b520 2024-04-25T01:22:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.046+0000 7f4b5effd640 1 -- 172.21.15.77:0/1698609351 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b6000a640 con 0x7f4b6815b520 2024-04-25T01:22:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.046+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/1698609351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b680b1570 con 0x7f4b6815b520 2024-04-25T01:22:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.046+0000 7f4b5effd640 1 -- 172.21.15.77:0/1698609351 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000aa30 con 0x7f4b6815b520 2024-04-25T01:22:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.046+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/1698609351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b680b1b30 con 0x7f4b6815b520 2024-04-25T01:22:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.047+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/1698609351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b68068180 con 0x7f4b6815b520 2024-04-25T01:22:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.047+0000 7f4b5effd640 1 -- 172.21.15.77:0/1698609351 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4b6001b070 con 0x7f4b6815b520 2024-04-25T01:22:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.047+0000 7f4b5effd640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b4806be30 0x7f4b4806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.047+0000 7f4b5effd640 1 -- 172.21.15.77:0/1698609351 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4b6008a740 con 0x7f4b6815b520 2024-04-25T01:22:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.048+0000 7f4b6659c640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b4806be30 0x7f4b4806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.048+0000 7f4b6659c640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b4806be30 0x7f4b4806e2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009ad0 tx=0x7f4b58008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.052+0000 7f4b5effd640 1 -- 172.21.15.77:0/1698609351 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4b68068180 con 0x7f4b6815b520 2024-04-25T01:22:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.225+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/1698609351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4b68068fb0 con 0x7f4b6815b520 2024-04-25T01:22:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.225+0000 7f4b5effd640 1 -- 172.21.15.77:0/1698609351 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f4b60058db0 con 0x7f4b6815b520 2024-04-25T01:22:16.206 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/1698609351 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b4806be30 msgr2=0x7f4b4806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b4806be30 0x7f4b4806e2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009ad0 tx=0x7f4b58008040 comp rx=0 tx=0).stop 2024-04-25T01:22:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/1698609351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b6815b520 msgr2=0x7f4b680b0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b6815b520 0x7f4b680b0d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000c780 tx=0x7f4b6000cc40 comp rx=0 tx=0).stop 2024-04-25T01:22:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/1698609351 shutdown_connections 2024-04-25T01:22:16.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b4806be30 0x7f4b4806e2f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b6815b520 0x7f4b680b0d90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b68158a70 0x7f4b681330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 --2- 172.21.15.77:0/1698609351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b680b37c0 0x7f4b6815d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/1698609351 >> 172.21.15.77:0/1698609351 conn(0x7f4b6806b720 msgr2=0x7f4b68141e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:16.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/1698609351 shutdown_connections 2024-04-25T01:22:16.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.227+0000 7f4b6cf4f640 1 -- 172.21.15.77:0/1698609351 wait complete. 2024-04-25T01:22:16.219 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:16.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.337+0000 7fec59e8c640 1 Processor -- start 2024-04-25T01:22:16.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.337+0000 7fec59e8c640 1 -- start start 2024-04-25T01:22:16.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.338+0000 7fec59e8c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fec540722e0 0x7fec5406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.338+0000 7fec59e8c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec5406e410 con 0x7fec5415fc40 2024-04-25T01:22:16.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.338+0000 7fec59e8c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec5406d110 con 0x7fec540cf4d0 2024-04-25T01:22:16.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.338+0000 7fec59e8c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fec54160aa0 con 0x7fec540722e0 2024-04-25T01:22:16.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.338+0000 7fec537fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fec540722e0 0x7fec5406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.338+0000 7fec52ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fec5415fc40 0x7fec540cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56622/0 (socket says 172.21.15.77:56622) 2024-04-25T01:22:16.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.338+0000 7fec52ffd640 1 -- 172.21.15.77:0/1821903224 learned_addr learned my addr 172.21.15.77:0/1821903224 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:16.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.339+0000 7fec537fe640 1 -- 172.21.15.77:0/1821903224 >> v1:172.21.15.121:6789/0 conn(0x7fec540cf4d0 legacy=0x7fec54070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.339+0000 7fec537fe640 1 -- 172.21.15.77:0/1821903224 >> v1:172.21.15.77:6789/0 conn(0x7fec5415fc40 legacy=0x7fec540cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.339+0000 7fec527fc640 1 -- 172.21.15.77:0/1821903224 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3927814317 0 0) 0x7fec5406e410 con 0x7fec5415fc40 2024-04-25T01:22:16.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.339+0000 7fec537fe640 1 -- 172.21.15.77:0/1821903224 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5406d3f0 con 0x7fec540722e0 2024-04-25T01:22:16.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.339+0000 7fec537fe640 1 --2- 172.21.15.77:0/1821903224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec540722e0 0x7fec5406ded0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fec440097a0 tx=0x7fec4402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9d4fa7913d100d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.339+0000 7fec527fc640 1 -- 172.21.15.77:0/1821903224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44037070 con 0x7fec540722e0 2024-04-25T01:22:16.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.339+0000 7fec527fc640 1 -- 172.21.15.77:0/1821903224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec44032440 con 0x7fec540722e0 2024-04-25T01:22:16.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.339+0000 7fec527fc640 1 -- 172.21.15.77:0/1821903224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44032810 con 0x7fec540722e0 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.340+0000 7fec59e8c640 1 -- 172.21.15.77:0/1821903224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec540722e0 msgr2=0x7fec5406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.340+0000 7fec59e8c640 1 --2- 172.21.15.77:0/1821903224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec540722e0 0x7fec5406ded0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fec440097a0 tx=0x7fec4402a1a0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.340+0000 7fec59e8c640 1 -- 172.21.15.77:0/1821903224 shutdown_connections 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.340+0000 7fec59e8c640 1 --2- 172.21.15.77:0/1821903224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec540722e0 0x7fec5406ded0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.340+0000 7fec59e8c640 1 -- 172.21.15.77:0/1821903224 >> 172.21.15.77:0/1821903224 conn(0x7fec5406b720 msgr2=0x7fec540b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.340+0000 7fec59e8c640 1 -- 172.21.15.77:0/1821903224 shutdown_connections 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.340+0000 7fec59e8c640 1 -- 172.21.15.77:0/1821903224 wait complete. 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.341+0000 7fec59e8c640 1 Processor -- start 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.341+0000 7fec59e8c640 1 -- start start 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec59e8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec540722e0 0x7fec54158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec59e8c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec540cf4d0 0x7fec54143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec59e8c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec5415fc40 0x7fec541435d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec59e8c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fec5406cec0 con 0x7fec540722e0 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec59e8c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fec54071260 con 0x7fec540cf4d0 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec59e8c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec540713e0 con 0x7fec5415fc40 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec540722e0 0x7fec54158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec540722e0 0x7fec54158c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56748/0 (socket says 172.21.15.77:56748) 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec537fe640 1 -- 172.21.15.77:0/3484002117 learned_addr learned my addr 172.21.15.77:0/3484002117 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec537fe640 1 -- 172.21.15.77:0/3484002117 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec5415fc40 msgr2=0x7fec541435d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec537fe640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec5415fc40 0x7fec541435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec537fe640 1 -- 172.21.15.77:0/3484002117 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec540cf4d0 msgr2=0x7fec54143090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec537fe640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec540cf4d0 0x7fec54143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec537fe640 1 -- 172.21.15.77:0/3484002117 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec541f2660 con 0x7fec540722e0 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.342+0000 7fec537fe640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec540722e0 0x7fec54158c90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fec440098c0 tx=0x7fec440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.343+0000 7fec3bfff640 1 -- 172.21.15.77:0/3484002117 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44037040 con 0x7fec540722e0 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.343+0000 7fec3bfff640 1 -- 172.21.15.77:0/3484002117 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec44042070 con 0x7fec540722e0 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.343+0000 7fec59e8c640 1 -- 172.21.15.77:0/3484002117 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec541e7350 con 0x7fec540722e0 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.343+0000 7fec3bfff640 1 -- 172.21.15.77:0/3484002117 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44030a70 con 0x7fec540722e0 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.343+0000 7fec59e8c640 1 -- 172.21.15.77:0/3484002117 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec541f2c00 con 0x7fec540722e0 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.344+0000 7fec59e8c640 1 -- 172.21.15.77:0/3484002117 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec54068fb0 con 0x7fec540722e0 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.344+0000 7fec3bfff640 1 -- 172.21.15.77:0/3484002117 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fec44030c10 con 0x7fec540722e0 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.344+0000 7fec3bfff640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec4006be60 0x7fec4006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.344+0000 7fec3bfff640 1 -- 172.21.15.77:0/3484002117 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fec440abbf0 con 0x7fec540722e0 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.344+0000 7fec52ffd640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec4006be60 0x7fec4006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.345+0000 7fec52ffd640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec4006be60 0x7fec4006e320 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fec48000c00 tx=0x7fec480079c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.348+0000 7fec3bfff640 1 -- 172.21.15.77:0/3484002117 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fec44045360 con 0x7fec540722e0 2024-04-25T01:22:16.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.521+0000 7fec59e8c640 1 -- 172.21.15.77:0/3484002117 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fec54072c90 con 0x7fec540722e0 2024-04-25T01:22:16.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.521+0000 7fec3bfff640 1 -- 172.21.15.77:0/3484002117 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7fec4407a260 con 0x7fec540722e0 2024-04-25T01:22:16.502 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 -- 172.21.15.77:0/3484002117 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec4006be60 msgr2=0x7fec4006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec4006be60 0x7fec4006e320 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fec48000c00 tx=0x7fec480079c0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 -- 172.21.15.77:0/3484002117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec540722e0 msgr2=0x7fec54158c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec540722e0 0x7fec54158c90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fec440098c0 tx=0x7fec440046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 -- 172.21.15.77:0/3484002117 shutdown_connections 2024-04-25T01:22:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec4006be60 0x7fec4006e320 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec5415fc40 0x7fec541435d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec540cf4d0 0x7fec54143090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 --2- 172.21.15.77:0/3484002117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec540722e0 0x7fec54158c90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 -- 172.21.15.77:0/3484002117 >> 172.21.15.77:0/3484002117 conn(0x7fec5406b720 msgr2=0x7fec5406e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 -- 172.21.15.77:0/3484002117 shutdown_connections 2024-04-25T01:22:16.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.523+0000 7fec59e8c640 1 -- 172.21.15.77:0/3484002117 wait complete. 2024-04-25T01:22:16.515 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.631+0000 7fd27e914640 1 Processor -- start 2024-04-25T01:22:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.631+0000 7fd27e914640 1 -- start start 2024-04-25T01:22:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.631+0000 7fd27e914640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd278143d20 0x7fd27815ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.631+0000 7fd27e914640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd27806ed60 0x7fd27815f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.631+0000 7fd27e914640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd27815f770 con 0x7fd278143940 2024-04-25T01:22:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.631+0000 7fd27e914640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fd278070e60 con 0x7fd27806ed60 2024-04-25T01:22:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.631+0000 7fd27e914640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd278071140 con 0x7fd278143d20 2024-04-25T01:22:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.631+0000 7fd27ceb1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd27806ed60 0x7fd27815f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.631+0000 7fd277fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd278143940 0x7fd27806e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36590/0 (socket says 172.21.15.77:36590) 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.632+0000 7fd277fff640 1 -- 172.21.15.77:0/1421507358 learned_addr learned my addr 172.21.15.77:0/1421507358 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.632+0000 7fd2777fe640 1 --2- 172.21.15.77:0/1421507358 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd278143d20 0x7fd27815ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.632+0000 7fd27ceb1640 1 -- 172.21.15.77:0/1421507358 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd278143d20 msgr2=0x7fd27815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.632+0000 7fd27ceb1640 1 --2- 172.21.15.77:0/1421507358 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd278143d20 0x7fd27815ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.632+0000 7fd27ceb1640 1 -- 172.21.15.77:0/1421507358 >> v1:172.21.15.77:6789/0 conn(0x7fd278143940 legacy=0x7fd27806e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.632+0000 7fd27ceb1640 1 -- 172.21.15.77:0/1421507358 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd278071420 con 0x7fd27806ed60 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.632+0000 7fd27ceb1640 1 --2- 172.21.15.77:0/1421507358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd27806ed60 0x7fd27815f230 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd270009dc0 tx=0x7fd27002c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9286f186689c0639 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.633+0000 7fd276ffd640 1 -- 172.21.15.77:0/1421507358 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270007740 con 0x7fd27806ed60 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.633+0000 7fd276ffd640 1 -- 172.21.15.77:0/1421507358 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2700078e0 con 0x7fd27806ed60 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.633+0000 7fd276ffd640 1 -- 172.21.15.77:0/1421507358 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270007cb0 con 0x7fd27806ed60 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.633+0000 7fd27e914640 1 -- 172.21.15.77:0/1421507358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd27806ed60 msgr2=0x7fd27815f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.633+0000 7fd27e914640 1 --2- 172.21.15.77:0/1421507358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd27806ed60 0x7fd27815f230 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd270009dc0 tx=0x7fd27002c790 comp rx=0 tx=0).stop 2024-04-25T01:22:16.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.633+0000 7fd27e914640 1 -- 172.21.15.77:0/1421507358 shutdown_connections 2024-04-25T01:22:16.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.633+0000 7fd27e914640 1 --2- 172.21.15.77:0/1421507358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd27806ed60 0x7fd27815f230 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.633+0000 7fd27e914640 1 --2- 172.21.15.77:0/1421507358 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd278143d20 0x7fd27815ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.633+0000 7fd27e914640 1 -- 172.21.15.77:0/1421507358 >> 172.21.15.77:0/1421507358 conn(0x7fd2780b1820 msgr2=0x7fd2780b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:16.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.633+0000 7fd27e914640 1 -- 172.21.15.77:0/1421507358 shutdown_connections 2024-04-25T01:22:16.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.633+0000 7fd27e914640 1 -- 172.21.15.77:0/1421507358 wait complete. 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.634+0000 7fd27e914640 1 Processor -- start 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.634+0000 7fd27e914640 1 -- start start 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd27e914640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27806ed60 0x7fd27815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd27e914640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd278143940 0x7fd2781426a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd27e914640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd278143d20 0x7fd278142be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd27e914640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd27806d6c0 con 0x7fd278143940 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd27e914640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd27806d540 con 0x7fd27806ed60 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd27e914640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd27806d840 con 0x7fd278143d20 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd2777fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd278143940 0x7fd2781426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd2777fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd278143940 0x7fd2781426a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45310/0 (socket says 172.21.15.77:45310) 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd2777fe640 1 -- 172.21.15.77:0/4198251485 learned_addr learned my addr 172.21.15.77:0/4198251485 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd27ceb1640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd278143d20 0x7fd278142be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd277fff640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27806ed60 0x7fd27815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd2777fe640 1 -- 172.21.15.77:0/4198251485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd278143d20 msgr2=0x7fd278142be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd2777fe640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd278143d20 0x7fd278142be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd2777fe640 1 -- 172.21.15.77:0/4198251485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27806ed60 msgr2=0x7fd27815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd2777fe640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27806ed60 0x7fd27815cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.635+0000 7fd2777fe640 1 -- 172.21.15.77:0/4198251485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2780ccff0 con 0x7fd278143940 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.636+0000 7fd2777fe640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd278143940 0x7fd2781426a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd26800d620 tx=0x7fd26800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.636+0000 7fd274ff9640 1 -- 172.21.15.77:0/4198251485 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26800b620 con 0x7fd278143940 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.636+0000 7fd274ff9640 1 -- 172.21.15.77:0/4198251485 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd268019070 con 0x7fd278143940 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.636+0000 7fd27e914640 1 -- 172.21.15.77:0/4198251485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2780cd290 con 0x7fd278143940 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.636+0000 7fd274ff9640 1 -- 172.21.15.77:0/4198251485 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26800ba00 con 0x7fd278143940 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.636+0000 7fd27e914640 1 -- 172.21.15.77:0/4198251485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2780cd850 con 0x7fd278143940 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.637+0000 7fd27e914640 1 -- 172.21.15.77:0/4198251485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd278068fb0 con 0x7fd278143940 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.637+0000 7fd274ff9640 1 -- 172.21.15.77:0/4198251485 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd26800bba0 con 0x7fd278143940 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.638+0000 7fd274ff9640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd26406be30 0x7fd26406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.638+0000 7fd274ff9640 1 -- 172.21.15.77:0/4198251485 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd26808b5b0 con 0x7fd278143940 2024-04-25T01:22:16.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.638+0000 7fd277fff640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd26406be30 0x7fd26406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.638+0000 7fd277fff640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd26406be30 0x7fd26406e2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd26c007fd0 tx=0x7fd26c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.642+0000 7fd274ff9640 1 -- 172.21.15.77:0/4198251485 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd268059c20 con 0x7fd278143940 2024-04-25T01:22:16.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.850+0000 7fd27e914640 1 -- 172.21.15.77:0/4198251485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd27806d150 con 0x7fd278143940 2024-04-25T01:22:16.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.851+0000 7fd274ff9640 1 -- 172.21.15.77:0/4198251485 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7fd268059520 con 0x7fd278143940 2024-04-25T01:22:16.832 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:16.832 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4267907621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4267907621},{"type":"v1","addr":"172.21.15.77:6837","nonce":4267907621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/493518847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":493518847},{"type":"v1","addr":"172.21.15.121:6833","nonce":493518847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:21:45.408154+0000","modified":"2024-04-25T01:22:06.554124+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6835/2810316639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2810316639},{"type":"v1","addr":"172.21.15.121:6835","nonce":2810316639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T01:22:16.832 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2024-04-25T01:22:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 -- 172.21.15.77:0/4198251485 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd26406be30 msgr2=0x7fd26406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd26406be30 0x7fd26406e2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd26c007fd0 tx=0x7fd26c009040 comp rx=0 tx=0).stop 2024-04-25T01:22:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 -- 172.21.15.77:0/4198251485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd278143940 msgr2=0x7fd2781426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd278143940 0x7fd2781426a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd26800d620 tx=0x7fd26800dae0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 -- 172.21.15.77:0/4198251485 shutdown_connections 2024-04-25T01:22:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd26406be30 0x7fd26406e2f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd278143940 0x7fd2781426a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd278143d20 0x7fd278142be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 --2- 172.21.15.77:0/4198251485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27806ed60 0x7fd27815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 -- 172.21.15.77:0/4198251485 >> 172.21.15.77:0/4198251485 conn(0x7fd2780b1820 msgr2=0x7fd2780725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 -- 172.21.15.77:0/4198251485 shutdown_connections 2024-04-25T01:22:16.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.853+0000 7fd27e914640 1 -- 172.21.15.77:0/4198251485 wait complete. 2024-04-25T01:22:16.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:16.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.960+0000 7f870f645640 1 Processor -- start 2024-04-25T01:22:16.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.960+0000 7f870f645640 1 -- start start 2024-04-25T01:22:16.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f870f645640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8708154ac0 0x7f870815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f870f645640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080cf2a0 0x7f87080711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f870f645640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8708154ea0 con 0x7f8708071730 2024-04-25T01:22:16.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f870f645640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f870806d540 con 0x7f8708154ac0 2024-04-25T01:22:16.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f870f645640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f870806d790 con 0x7f87080cf2a0 2024-04-25T01:22:16.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f870dbe2640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8708071730 0x7f870806e550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36602/0 (socket says 172.21.15.77:36602) 2024-04-25T01:22:16.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f870dbe2640 1 -- 172.21.15.77:0/2296085707 learned_addr learned my addr 172.21.15.77:0/2296085707 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:16.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f870cbe0640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080cf2a0 0x7f87080711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f870d3e1640 1 --2- 172.21.15.77:0/2296085707 >> v2:172.21.15.121:3300/0 conn(0x7f8708154ac0 0x7f870815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1658757189 0 0) 0x7f8708154ea0 con 0x7f8708071730 2024-04-25T01:22:16.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86f4003520 con 0x7f8708071730 2024-04-25T01:22:16.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1819732337 0 0) 0x7f86f4003520 con 0x7f8708071730 2024-04-25T01:22:16.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080cf2a0 msgr2=0x7f87080711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:16.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080cf2a0 0x7f87080711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 >> v2:172.21.15.121:3300/0 conn(0x7f8708154ac0 msgr2=0x7f870815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 --2- 172.21.15.77:0/2296085707 >> v2:172.21.15.121:3300/0 conn(0x7f8708154ac0 0x7f870815b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f870806da70 con 0x7f8708071730 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f87000019f0 con 0x7f8708071730 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 >> v1:172.21.15.77:6789/0 conn(0x7f8708071730 legacy=0x7f870806e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080cf2a0 0x7f86f40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f4003f30 0x7f86f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86f4004d40 0x7f86f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f870806d540 con 0x7f86f4003f30 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f870806d790 con 0x7f86f4004d40 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f86f400cad0 con 0x7f87080cf2a0 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.961+0000 7f870cbe0640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f4003f30 0x7f86f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.962+0000 7f870d3e1640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080cf2a0 0x7f86f40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.962+0000 7f870dbe2640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86f4004d40 0x7f86f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.962+0000 7f870cbe0640 1 -- 172.21.15.77:0/2296085707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080cf2a0 msgr2=0x7f86f40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.962+0000 7f870cbe0640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080cf2a0 0x7f86f40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.962+0000 7f870cbe0640 1 -- 172.21.15.77:0/2296085707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86f4004d40 msgr2=0x7f86f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.962+0000 7f870cbe0640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86f4004d40 0x7f86f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.962+0000 7f870cbe0640 1 -- 172.21.15.77:0/2296085707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f4003520 con 0x7f86f4003f30 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.962+0000 7f870cbe0640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f4003f30 0x7f86f4004800 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8704009ac0 tx=0x7f86f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c115eba6082b69 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.962+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704036070 con 0x7f86f4003f30 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.963+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8704031400 con 0x7f86f4003f30 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.963+0000 7f86fbfff640 1 -- 172.21.15.77:0/2296085707 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704009690 con 0x7f86f4003f30 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.963+0000 7f870f645640 1 -- 172.21.15.77:0/2296085707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f4003f30 msgr2=0x7f86f4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.963+0000 7f870f645640 1 --2- 172.21.15.77:0/2296085707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f4003f30 0x7f86f4004800 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8704009ac0 tx=0x7f86f4005830 comp rx=0 tx=0).stop 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.963+0000 7f870dbe2640 1 -- 172.21.15.77:0/2296085707 reap_dead start 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.963+0000 7f870f645640 1 -- 172.21.15.77:0/2296085707 shutdown_connections 2024-04-25T01:22:16.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.963+0000 7f870f645640 1 -- 172.21.15.77:0/2296085707 >> 172.21.15.77:0/2296085707 conn(0x7f87080b1820 msgr2=0x7f87080b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:16.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.963+0000 7f870f645640 1 -- 172.21.15.77:0/2296085707 shutdown_connections 2024-04-25T01:22:16.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.963+0000 7f870f645640 1 -- 172.21.15.77:0/2296085707 wait complete. 2024-04-25T01:22:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.964+0000 7f870f645640 1 Processor -- start 2024-04-25T01:22:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.964+0000 7f870f645640 1 -- start start 2024-04-25T01:22:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870f645640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86f4003f30 0x7f8708148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870f645640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87081590f0 0x7f8708159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870f645640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080c8ea0 0x7f87080c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870f645640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f87080cf720 con 0x7f87081590f0 2024-04-25T01:22:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870f645640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f870806e030 con 0x7f86f4003f30 2024-04-25T01:22:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870f645640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f870806ec60 con 0x7f87080c8ea0 2024-04-25T01:22:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870cbe0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87081590f0 0x7f8708159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870cbe0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87081590f0 0x7f8708159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45334/0 (socket says 172.21.15.77:45334) 2024-04-25T01:22:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870cbe0640 1 -- 172.21.15.77:0/243711737 learned_addr learned my addr 172.21.15.77:0/243711737 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870d3e1640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86f4003f30 0x7f8708148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870dbe2640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080c8ea0 0x7f87080c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870cbe0640 1 -- 172.21.15.77:0/243711737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080c8ea0 msgr2=0x7f87080c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870cbe0640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080c8ea0 0x7f87080c9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.965+0000 7f870cbe0640 1 -- 172.21.15.77:0/243711737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86f4003f30 msgr2=0x7f8708148f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.966+0000 7f870cbe0640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86f4003f30 0x7f8708148f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.966+0000 7f870cbe0640 1 -- 172.21.15.77:0/243711737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87080c9860 con 0x7f87081590f0 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.966+0000 7f870cbe0640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87081590f0 0x7f8708159550 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8704009be0 tx=0x7f8704004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.966+0000 7f86f9ffb640 1 -- 172.21.15.77:0/243711737 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704040020 con 0x7f87081590f0 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.966+0000 7f86f9ffb640 1 -- 172.21.15.77:0/243711737 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8704003c60 con 0x7f87081590f0 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.966+0000 7f86f9ffb640 1 -- 172.21.15.77:0/243711737 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704036040 con 0x7f87081590f0 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.966+0000 7f870f645640 1 -- 172.21.15.77:0/243711737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87080c9b20 con 0x7f87081590f0 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.966+0000 7f870f645640 1 -- 172.21.15.77:0/243711737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87080ca0e0 con 0x7f87081590f0 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.967+0000 7f870f645640 1 -- 172.21.15.77:0/243711737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8708068fb0 con 0x7f87081590f0 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.967+0000 7f86f9ffb640 1 -- 172.21.15.77:0/243711737 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8704043020 con 0x7f87081590f0 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.968+0000 7f86f9ffb640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86ec06be30 0x7f86ec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.968+0000 7f86f9ffb640 1 -- 172.21.15.77:0/243711737 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f87040ab240 con 0x7f87081590f0 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.968+0000 7f870d3e1640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86ec06be30 0x7f86ec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.968+0000 7f870d3e1640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86ec06be30 0x7f86ec06e2f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009b80 tx=0x7f86fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:16.972+0000 7f86f9ffb640 1 -- 172.21.15.77:0/243711737 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f870404a070 con 0x7f87081590f0 2024-04-25T01:22:17.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.151+0000 7f870f645640 1 -- 172.21.15.77:0/243711737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f870815b3d0 con 0x7f87081590f0 2024-04-25T01:22:17.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.152+0000 7f86f9ffb640 1 -- 172.21.15.77:0/243711737 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f87040044e0 con 0x7f87081590f0 2024-04-25T01:22:17.133 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:17.133 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:21:50.465449+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.154+0000 7f870f645640 1 -- 172.21.15.77:0/243711737 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86ec06be30 msgr2=0x7f86ec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.154+0000 7f870f645640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86ec06be30 0x7f86ec06e2f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009b80 tx=0x7f86fc008040 comp rx=0 tx=0).stop 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.154+0000 7f870f645640 1 -- 172.21.15.77:0/243711737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87081590f0 msgr2=0x7f8708159550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.154+0000 7f870f645640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87081590f0 0x7f8708159550 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8704009be0 tx=0x7f8704004730 comp rx=0 tx=0).stop 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.155+0000 7f870f645640 1 -- 172.21.15.77:0/243711737 shutdown_connections 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.155+0000 7f870f645640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86ec06be30 0x7f86ec06e2f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.155+0000 7f870f645640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87081590f0 0x7f8708159550 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.155+0000 7f870f645640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87080c8ea0 0x7f87080c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.155+0000 7f870f645640 1 --2- 172.21.15.77:0/243711737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86f4003f30 0x7f8708148f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.155+0000 7f870f645640 1 -- 172.21.15.77:0/243711737 >> 172.21.15.77:0/243711737 conn(0x7f87080b1820 msgr2=0x7f8708153b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.155+0000 7f870f645640 1 -- 172.21.15.77:0/243711737 shutdown_connections 2024-04-25T01:22:17.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.155+0000 7f870f645640 1 -- 172.21.15.77:0/243711737 wait complete. 2024-04-25T01:22:17.147 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:17.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.263+0000 7fde7f3ce640 1 Processor -- start 2024-04-25T01:22:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.263+0000 7fde7f3ce640 1 -- start start 2024-04-25T01:22:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7f3ce640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde780d0010 0x7fde780d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7f3ce640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fde780af220 0x7fde780af670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7f3ce640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde780703e0 con 0x7fde7815f350 2024-04-25T01:22:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7f3ce640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fde780aed90 con 0x7fde780af220 2024-04-25T01:22:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7f3ce640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fde780aea60 con 0x7fde780d0010 2024-04-25T01:22:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7d16a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fde7815f350 0x7fde7806fcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36610/0 (socket says 172.21.15.77:36610) 2024-04-25T01:22:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7d16a640 1 -- 172.21.15.77:0/768502722 learned_addr learned my addr 172.21.15.77:0/768502722 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7d96b640 1 --2- 172.21.15.77:0/768502722 >> v2:172.21.15.121:3300/0 conn(0x7fde780af220 0x7fde780af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7c969640 1 --2- 172.21.15.77:0/768502722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde780d0010 0x7fde780d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7c969640 1 -- 172.21.15.77:0/768502722 >> v2:172.21.15.121:3300/0 conn(0x7fde780af220 msgr2=0x7fde780af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7c969640 1 --2- 172.21.15.77:0/768502722 >> v2:172.21.15.121:3300/0 conn(0x7fde780af220 0x7fde780af670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7c969640 1 -- 172.21.15.77:0/768502722 >> v1:172.21.15.77:6789/0 conn(0x7fde7815f350 legacy=0x7fde7806fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7c969640 1 -- 172.21.15.77:0/768502722 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde780d0970 con 0x7fde780d0010 2024-04-25T01:22:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.264+0000 7fde7d96b640 1 --2- 172.21.15.77:0/768502722 >> v2:172.21.15.121:3300/0 conn(0x7fde780af220 0x7fde780af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:17.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.265+0000 7fde7c969640 1 --2- 172.21.15.77:0/768502722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde780d0010 0x7fde780d0430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fde680097a0 tx=0x7fde6802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5afdb95ba3330c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.265+0000 7fde6ffff640 1 -- 172.21.15.77:0/768502722 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde68037070 con 0x7fde780d0010 2024-04-25T01:22:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.265+0000 7fde6ffff640 1 -- 172.21.15.77:0/768502722 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde68032440 con 0x7fde780d0010 2024-04-25T01:22:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.265+0000 7fde6ffff640 1 -- 172.21.15.77:0/768502722 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde68032810 con 0x7fde780d0010 2024-04-25T01:22:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.266+0000 7fde7f3ce640 1 -- 172.21.15.77:0/768502722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde780d0010 msgr2=0x7fde780d0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.266+0000 7fde7f3ce640 1 --2- 172.21.15.77:0/768502722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde780d0010 0x7fde780d0430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fde680097a0 tx=0x7fde6802a180 comp rx=0 tx=0).stop 2024-04-25T01:22:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.266+0000 7fde7f3ce640 1 -- 172.21.15.77:0/768502722 shutdown_connections 2024-04-25T01:22:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.266+0000 7fde7f3ce640 1 --2- 172.21.15.77:0/768502722 >> v2:172.21.15.121:3300/0 conn(0x7fde780af220 0x7fde780af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.266+0000 7fde7f3ce640 1 --2- 172.21.15.77:0/768502722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde780d0010 0x7fde780d0430 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.266+0000 7fde7f3ce640 1 -- 172.21.15.77:0/768502722 >> 172.21.15.77:0/768502722 conn(0x7fde7809aa70 msgr2=0x7fde7809ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.266+0000 7fde7f3ce640 1 -- 172.21.15.77:0/768502722 shutdown_connections 2024-04-25T01:22:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.266+0000 7fde7f3ce640 1 -- 172.21.15.77:0/768502722 wait complete. 2024-04-25T01:22:17.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.267+0000 7fde7f3ce640 1 Processor -- start 2024-04-25T01:22:17.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.267+0000 7fde7f3ce640 1 -- start start 2024-04-25T01:22:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7f3ce640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde780af220 0x7fde780ce320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7f3ce640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde780d0010 0x7fde780ce860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7f3ce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde7815f350 0x7fde780b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7f3ce640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fde78070c10 con 0x7fde7815f350 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7f3ce640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fde78070930 con 0x7fde780af220 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7f3ce640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fde7815f730 con 0x7fde780d0010 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7d96b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde7815f350 0x7fde780b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7d96b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde7815f350 0x7fde780b4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45344/0 (socket says 172.21.15.77:45344) 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7d96b640 1 -- 172.21.15.77:0/3334852079 learned_addr learned my addr 172.21.15.77:0/3334852079 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7c969640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde780d0010 0x7fde780ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7d16a640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde780af220 0x7fde780ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7d96b640 1 -- 172.21.15.77:0/3334852079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde780d0010 msgr2=0x7fde780ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7d96b640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde780d0010 0x7fde780ce860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7d96b640 1 -- 172.21.15.77:0/3334852079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde780af220 msgr2=0x7fde780ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7d96b640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde780af220 0x7fde780ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7d96b640 1 -- 172.21.15.77:0/3334852079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde780c8f30 con 0x7fde7815f350 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde7d96b640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde7815f350 0x7fde780b4820 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fde70009f00 tx=0x7fde7000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.268+0000 7fde6dffb640 1 -- 172.21.15.77:0/3334852079 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde70018030 con 0x7fde7815f350 2024-04-25T01:22:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.269+0000 7fde6dffb640 1 -- 172.21.15.77:0/3334852079 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde7000ce50 con 0x7fde7815f350 2024-04-25T01:22:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.269+0000 7fde6dffb640 1 -- 172.21.15.77:0/3334852079 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde70003bb0 con 0x7fde7815f350 2024-04-25T01:22:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.269+0000 7fde7f3ce640 1 -- 172.21.15.77:0/3334852079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde780c9210 con 0x7fde7815f350 2024-04-25T01:22:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.269+0000 7fde7f3ce640 1 -- 172.21.15.77:0/3334852079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde780b4d60 con 0x7fde7815f350 2024-04-25T01:22:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.270+0000 7fde7f3ce640 1 -- 172.21.15.77:0/3334852079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde780cb6f0 con 0x7fde7815f350 2024-04-25T01:22:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.270+0000 7fde6dffb640 1 -- 172.21.15.77:0/3334852079 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fde7001b070 con 0x7fde7815f350 2024-04-25T01:22:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.270+0000 7fde6dffb640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde5c06be30 0x7fde5c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.270+0000 7fde6dffb640 1 -- 172.21.15.77:0/3334852079 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fde7008ba70 con 0x7fde7815f350 2024-04-25T01:22:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.270+0000 7fde7d16a640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde5c06be30 0x7fde5c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.271+0000 7fde7d16a640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde5c06be30 0x7fde5c06e2f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fde780b7b50 tx=0x7fde74007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:17.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.274+0000 7fde6dffb640 1 -- 172.21.15.77:0/3334852079 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fde70010040 con 0x7fde7815f350 2024-04-25T01:22:17.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.481+0000 7fde7f3ce640 1 -- 172.21.15.77:0/3334852079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde78068180 con 0x7fde7815f350 2024-04-25T01:22:17.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.481+0000 7fde6dffb640 1 -- 172.21.15.77:0/3334852079 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7fde70059b20 con 0x7fde7815f350 2024-04-25T01:22:17.462 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:17.462 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4267907621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4267907621},{"type":"v1","addr":"172.21.15.77:6837","nonce":4267907621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/493518847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":493518847},{"type":"v1","addr":"172.21.15.121:6833","nonce":493518847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:21:45.408154+0000","modified":"2024-04-25T01:22:06.554124+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6835/2810316639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2810316639},{"type":"v1","addr":"172.21.15.121:6835","nonce":2810316639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T01:22:17.463 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2024-04-25T01:22:17.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 -- 172.21.15.77:0/3334852079 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde5c06be30 msgr2=0x7fde5c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde5c06be30 0x7fde5c06e2f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fde780b7b50 tx=0x7fde74007e80 comp rx=0 tx=0).stop 2024-04-25T01:22:17.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 -- 172.21.15.77:0/3334852079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde7815f350 msgr2=0x7fde780b4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde7815f350 0x7fde780b4820 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fde70009f00 tx=0x7fde7000c490 comp rx=0 tx=0).stop 2024-04-25T01:22:17.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 -- 172.21.15.77:0/3334852079 shutdown_connections 2024-04-25T01:22:17.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde5c06be30 0x7fde5c06e2f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde7815f350 0x7fde780b4820 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde780d0010 0x7fde780ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 --2- 172.21.15.77:0/3334852079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde780af220 0x7fde780ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 -- 172.21.15.77:0/3334852079 >> 172.21.15.77:0/3334852079 conn(0x7fde7809aa70 msgr2=0x7fde780af650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:17.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 -- 172.21.15.77:0/3334852079 shutdown_connections 2024-04-25T01:22:17.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.483+0000 7fde7f3ce640 1 -- 172.21.15.77:0/3334852079 wait complete. 2024-04-25T01:22:17.878 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:21:45.408154+0000', 'modified': '2024-04-25T01:22:06.554124+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6835/2810316639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6834', 'nonce': 2810316639}, {'type': 'v1', 'addr': '172.21.15.121:6835', 'nonce': 2810316639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4294, 'qdb_cluster': [4294]} 2024-04-25T01:22:17.878 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:22:17.879 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T01:22:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.997+0000 7f9e8e590640 1 Processor -- start 2024-04-25T01:22:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.997+0000 7f9e8e590640 1 -- start start 2024-04-25T01:22:17.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.998+0000 7f9e8e590640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880d4a10 0x7f9e880cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.998+0000 7f9e8e590640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9e880b3ee0 0x7f9e880b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.998+0000 7f9e8e590640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e880d1e90 con 0x7f9e8815e4c0 2024-04-25T01:22:17.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.998+0000 7f9e8e590640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9e880b54a0 con 0x7f9e880b3ee0 2024-04-25T01:22:17.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.998+0000 7f9e8e590640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9e880d2130 con 0x7f9e880d4a10 2024-04-25T01:22:17.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.998+0000 7f9e87fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9e8815e4c0 0x7f9e880d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36624/0 (socket says 172.21.15.77:36624) 2024-04-25T01:22:17.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.998+0000 7f9e87fff640 1 -- 172.21.15.77:0/3393474066 learned_addr learned my addr 172.21.15.77:0/3393474066 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:17.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.998+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/3393474066 >> v2:172.21.15.121:3300/0 conn(0x7f9e880b3ee0 0x7f9e880b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.998+0000 7f9e877fe640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880d4a10 0x7f9e880cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2948806417 0 0) 0x7f9e880d1e90 con 0x7f9e8815e4c0 2024-04-25T01:22:17.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e70003520 con 0x7f9e8815e4c0 2024-04-25T01:22:17.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1073281193 0 0) 0x7f9e70003520 con 0x7f9e8815e4c0 2024-04-25T01:22:17.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880d4a10 msgr2=0x7f9e880cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:17.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880d4a10 0x7f9e880cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 >> v2:172.21.15.121:3300/0 conn(0x7f9e880b3ee0 msgr2=0x7f9e880b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:17.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 --2- 172.21.15.77:0/3393474066 >> v2:172.21.15.121:3300/0 conn(0x7f9e880b3ee0 0x7f9e880b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e880b5780 con 0x7f9e8815e4c0 2024-04-25T01:22:17.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f9e78001670 con 0x7f9e8815e4c0 2024-04-25T01:22:17.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 >> v1:172.21.15.77:6789/0 conn(0x7f9e8815e4c0 legacy=0x7f9e880d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880d4a10 0x7f9e700039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e70003f30 0x7f9e70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 0x7f9e700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9e880b54a0 con 0x7f9e70004d40 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9e880d2130 con 0x7f9e70003f30 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9e7000ba10 con 0x7f9e880d4a10 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 0x7f9e700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e87fff640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880d4a10 0x7f9e700039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e877fe640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e70003f30 0x7f9e70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e8cb2d640 1 -- 172.21.15.77:0/3393474066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880d4a10 msgr2=0x7f9e700039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880d4a10 0x7f9e700039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e8cb2d640 1 -- 172.21.15.77:0/3393474066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e70003f30 msgr2=0x7f9e70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e70003f30 0x7f9e70004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:17.999+0000 7f9e8cb2d640 1 -- 172.21.15.77:0/3393474066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e70003520 con 0x7f9e70004d40 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.000+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 0x7f9e700051c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000e170 tx=0x7f9e70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae59f85c7051fc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.000+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80034020 con 0x7f9e70004d40 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.000+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e80030430 con 0x7f9e70004d40 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.000+0000 7f9e86ffd640 1 -- 172.21.15.77:0/3393474066 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80009c10 con 0x7f9e70004d40 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.001+0000 7f9e8e590640 1 -- 172.21.15.77:0/3393474066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 msgr2=0x7f9e700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.001+0000 7f9e8e590640 1 --2- 172.21.15.77:0/3393474066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 0x7f9e700051c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000e170 tx=0x7f9e70005830 comp rx=0 tx=0).stop 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.001+0000 7f9e8cb2d640 1 -- 172.21.15.77:0/3393474066 reap_dead start 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.001+0000 7f9e8e590640 1 -- 172.21.15.77:0/3393474066 shutdown_connections 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.001+0000 7f9e8e590640 1 -- 172.21.15.77:0/3393474066 >> 172.21.15.77:0/3393474066 conn(0x7f9e880d3ef0 msgr2=0x7f9e880d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.001+0000 7f9e8e590640 1 -- 172.21.15.77:0/3393474066 shutdown_connections 2024-04-25T01:22:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.001+0000 7f9e8e590640 1 -- 172.21.15.77:0/3393474066 wait complete. 2024-04-25T01:22:17.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.002+0000 7f9e8e590640 1 Processor -- start 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.002+0000 7f9e8e590640 1 -- start start 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e8e590640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 0x7f9e88159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e8e590640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e8815a4f0 0x7f9e880c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e8e590640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880ca160 0x7f9e881e6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e8e590640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9e880b30a0 con 0x7f9e70004d40 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e8e590640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9e880b2f20 con 0x7f9e8815a4f0 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e8e590640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9e8815e8a0 con 0x7f9e880ca160 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e87fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 0x7f9e88159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e87fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 0x7f9e88159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45360/0 (socket says 172.21.15.77:45360) 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e87fff640 1 -- 172.21.15.77:0/1955787017 learned_addr learned my addr 172.21.15.77:0/1955787017 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e877fe640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e8815a4f0 0x7f9e880c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880ca160 0x7f9e881e6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e87fff640 1 -- 172.21.15.77:0/1955787017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880ca160 msgr2=0x7f9e881e6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e87fff640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880ca160 0x7f9e881e6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e87fff640 1 -- 172.21.15.77:0/1955787017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e8815a4f0 msgr2=0x7f9e880c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e87fff640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e8815a4f0 0x7f9e880c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e87fff640 1 -- 172.21.15.77:0/1955787017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e880c88a0 con 0x7f9e70004d40 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880ca160 0x7f9e881e6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e87fff640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 0x7f9e88159cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9e78007310 tx=0x7f9e78007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e84ff9640 1 -- 172.21.15.77:0/1955787017 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7800f9c0 con 0x7f9e70004d40 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e84ff9640 1 -- 172.21.15.77:0/1955787017 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e78018070 con 0x7f9e70004d40 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e8e590640 1 -- 172.21.15.77:0/1955787017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e880c8b80 con 0x7f9e70004d40 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.003+0000 7f9e84ff9640 1 -- 172.21.15.77:0/1955787017 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7800fda0 con 0x7f9e70004d40 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.004+0000 7f9e8e590640 1 -- 172.21.15.77:0/1955787017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e880c9140 con 0x7f9e70004d40 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.005+0000 7f9e84ff9640 1 -- 172.21.15.77:0/1955787017 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9e7801d460 con 0x7f9e70004d40 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.005+0000 7f9e84ff9640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e7406be30 0x7f9e7406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.005+0000 7f9e84ff9640 1 -- 172.21.15.77:0/1955787017 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9e7808b280 con 0x7f9e70004d40 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.005+0000 7f9e877fe640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e7406be30 0x7f9e7406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.006+0000 7f9e877fe640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e7406be30 0x7f9e7406e2f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9e881f74e0 tx=0x7f9e7c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:17.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.006+0000 7f9e9137c740 1 Processor -- start 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.006+0000 7f9e9137c740 1 -- start start 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b26adb60 0x5609b26604f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 0x5609b2690000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b2665350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e9137c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5609b25c8f00 con 0x5609b2690540 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e9137c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5609b2571750 con 0x5609b26adb60 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e9137c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5609b25c4ec0 con 0x5609b2690c80 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e8cb2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b2665350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e8cb2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b2665350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45374/0 (socket says 172.21.15.77:45374) 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e8cb2d640 1 -- 172.21.15.77:0/3899805354 learned_addr learned my addr 172.21.15.77:0/3899805354 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e877fe640 1 --2- 172.21.15.77:0/3899805354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 0x5609b2690000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e87fff640 1 --2- 172.21.15.77:0/3899805354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b26adb60 0x5609b26604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e8cb2d640 1 -- 172.21.15.77:0/3899805354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 msgr2=0x5609b2690000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:17.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/3899805354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 0x5609b2690000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e8cb2d640 1 -- 172.21.15.77:0/3899805354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b26adb60 msgr2=0x5609b26604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/3899805354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b26adb60 0x5609b26604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e8cb2d640 1 -- 172.21.15.77:0/3899805354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609b22dd2d0 con 0x5609b2690540 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.007+0000 7f9e877fe640 1 --2- 172.21.15.77:0/3899805354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 0x5609b2690000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/3899805354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b2665350 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000e290 tx=0x7f9e8002d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3520acb7171951 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e6dffb640 1 -- 172.21.15.77:0/3899805354 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80034040 con 0x5609b2690540 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e6dffb640 1 -- 172.21.15.77:0/3899805354 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e80030430 con 0x5609b2690540 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e6dffb640 1 -- 172.21.15.77:0/3899805354 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80003990 con 0x5609b2690540 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e9137c740 1 -- 172.21.15.77:0/3899805354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 msgr2=0x5609b2665350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e9137c740 1 --2- 172.21.15.77:0/3899805354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b2665350 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000e290 tx=0x7f9e8002d000 comp rx=0 tx=0).stop 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e9137c740 1 -- 172.21.15.77:0/3899805354 shutdown_connections 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e9137c740 1 --2- 172.21.15.77:0/3899805354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b2665350 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e9137c740 1 --2- 172.21.15.77:0/3899805354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 0x5609b2690000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e9137c740 1 --2- 172.21.15.77:0/3899805354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b26adb60 0x5609b26604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e9137c740 1 -- 172.21.15.77:0/3899805354 >> 172.21.15.77:0/3899805354 conn(0x5609b2662760 msgr2=0x5609b26ab000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e9137c740 1 -- 172.21.15.77:0/3899805354 shutdown_connections 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.008+0000 7f9e9137c740 1 -- 172.21.15.77:0/3899805354 wait complete. 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.009+0000 7f9e9137c740 1 -- start start 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.009+0000 7f9e9137c740 1 Processor -- start 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690c80 0x5609b26bddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b26adb60 0x5609b2692710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2692c50 0x5609b26ac130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e9137c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5609b2664910 con 0x5609b2690c80 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e9137c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5609b2584cf0 con 0x5609b2692c50 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e9137c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5609b2663d00 con 0x5609b26adb60 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e9137c740 10 client.0 _ll_register_callbacks cb 0x5609b25846e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e87fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690c80 0x5609b26bddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e87fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690c80 0x5609b26bddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45378/0 (socket says 172.21.15.77:45378) 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e87fff640 1 -- 172.21.15.77:0/1445268907 learned_addr learned my addr 172.21.15.77:0/1445268907 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e877fe640 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b26adb60 0x5609b2692710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2692c50 0x5609b26ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e87fff640 1 -- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b26adb60 msgr2=0x5609b2692710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e87fff640 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b26adb60 0x5609b2692710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e87fff640 1 -- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2692c50 msgr2=0x5609b26ac130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e87fff640 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2692c50 0x5609b26ac130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e87fff640 1 -- 172.21.15.77:0/1445268907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609b23a76b0 con 0x5609b2690c80 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2692c50 0x5609b26ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e87fff640 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690c80 0x5609b26bddd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9e78059210 tx=0x7f9e78059a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7599641f7c94d07a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e66ffd640 1 -- 172.21.15.77:0/1445268907 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e78082e60 con 0x5609b2690c80 2024-04-25T01:22:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e66ffd640 1 -- 172.21.15.77:0/1445268907 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9e78071170 con 0x5609b2690c80 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.010+0000 7f9e9137c740 1 -- 172.21.15.77:0/1445268907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e80030430 con 0x5609b2690c80 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e66ffd640 1 -- 172.21.15.77:0/1445268907 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f9e78061e60 con 0x5609b2690c80 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e9137c740 10 client.4830 fetch_fsmap learned FSMap version 9 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e9137c740 1 -- 172.21.15.77:0/1445268907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x5609b26bc130 con 0x5609b2690c80 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e66ffd640 1 -- 172.21.15.77:0/1445268907 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e80030430 con 0x5609b2690c80 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e66ffd640 1 -- 172.21.15.77:0/1445268907 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f9e7805d150 con 0x5609b2690c80 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e9137c740 10 client.4830 fetch_fsmap finished waiting for FSMap version 9 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e9137c740 10 client.4830 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e9137c740 20 client.4830 populate_metadata read hostname 'smithi077' 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e9137c740 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x5609b26915f0 0x5609b26bb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e9137c740 4 client.4830 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e9137c740 1 -- 172.21.15.77:0/1445268907 --> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609b25d6440 con 0x5609b26915f0 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e877fe640 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x5609b26915f0 0x5609b26bb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e877fe640 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x5609b26915f0 0x5609b26bb780 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87d079c5f095dc17 server_cookie=dbc27c3f0bcf991e in_seq=0 out_seq=0 2024-04-25T01:22:18.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.011+0000 7f9e66ffd640 10 client.4830 ms_handle_connect on v2:172.21.15.121:6834/2810316639 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e66ffd640 1 -- 172.21.15.77:0/1445268907 <== mds.0 v2:172.21.15.121:6834/2810316639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5609b25d6440 con 0x5609b26915f0 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e66ffd640 10 client.4830 handle_command_reply: tid=0 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e9137c740 1 client.4830 shutdown 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e9137c740 1 -- 172.21.15.77:0/1445268907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690c80 msgr2=0x5609b26bddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e9137c740 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690c80 0x5609b26bddd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9e78059210 tx=0x7f9e78059a70 comp rx=0 tx=0).stop 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e9137c740 1 -- 172.21.15.77:0/1445268907 shutdown_connections 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e9137c740 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x5609b26915f0 0x5609b26bb780 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e9137c740 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2692c50 0x5609b26ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e66ffd640 0 client.4830 ms_handle_reset on v2:172.21.15.121:6834/2810316639 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e9137c740 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b26adb60 0x5609b2692710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e9137c740 1 --2- 172.21.15.77:0/1445268907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690c80 0x5609b26bddd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e9137c740 1 -- 172.21.15.77:0/1445268907 >> 172.21.15.77:0/1445268907 conn(0x5609b2690540 msgr2=0x5609b265fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:18.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e9137c740 1 -- 172.21.15.77:0/1445268907 shutdown_connections 2024-04-25T01:22:18.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.012+0000 7f9e9137c740 1 -- 172.21.15.77:0/1445268907 wait complete. 2024-04-25T01:22:18.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.013+0000 7f9e9137c740 20 client.4830 trim_cache size 0 max 16384 2024-04-25T01:22:18.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.030+0000 7f9e9137c740 1 Processor -- start 2024-04-25T01:22:18.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.030+0000 7f9e9137c740 1 -- start start 2024-04-25T01:22:18.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b26aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 0x5609b26921e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2692c50 0x5609b26bc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e9137c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9e882010d0 con 0x5609b2690540 2024-04-25T01:22:18.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e9137c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9e88200430 con 0x5609b2692c50 2024-04-25T01:22:18.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e9137c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9e88200a80 con 0x5609b2690c80 2024-04-25T01:22:18.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e87fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b26aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e87fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b26aa4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45394/0 (socket says 172.21.15.77:45394) 2024-04-25T01:22:18.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e87fff640 1 -- 172.21.15.77:0/1858394878 learned_addr learned my addr 172.21.15.77:0/1858394878 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:18.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e877fe640 1 --2- 172.21.15.77:0/1858394878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 0x5609b26921e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/1858394878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2692c50 0x5609b26bc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e87fff640 1 -- 172.21.15.77:0/1858394878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 msgr2=0x5609b26921e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e87fff640 1 --2- 172.21.15.77:0/1858394878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 0x5609b26921e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e87fff640 1 -- 172.21.15.77:0/1858394878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2692c50 msgr2=0x5609b26bc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e87fff640 1 --2- 172.21.15.77:0/1858394878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2692c50 0x5609b26bc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e87fff640 1 -- 172.21.15.77:0/1858394878 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609b26ac4a0 con 0x5609b2690540 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.031+0000 7f9e877fe640 1 --2- 172.21.15.77:0/1858394878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 0x5609b26921e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e87fff640 1 --2- 172.21.15.77:0/1858394878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b26aa4c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9e78001050 tx=0x7f9e78009070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a708a603b7269a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e6d7fa640 1 -- 172.21.15.77:0/1858394878 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e78075020 con 0x5609b2690540 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e6d7fa640 1 -- 172.21.15.77:0/1858394878 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e780891f0 con 0x5609b2690540 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e6d7fa640 1 -- 172.21.15.77:0/1858394878 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e78082e60 con 0x5609b2690540 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e9137c740 1 -- 172.21.15.77:0/1858394878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 msgr2=0x5609b26aa4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e9137c740 1 --2- 172.21.15.77:0/1858394878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b26aa4c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9e78001050 tx=0x7f9e78009070 comp rx=0 tx=0).stop 2024-04-25T01:22:18.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e9137c740 1 -- 172.21.15.77:0/1858394878 shutdown_connections 2024-04-25T01:22:18.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e9137c740 1 --2- 172.21.15.77:0/1858394878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2692c50 0x5609b26bc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e9137c740 1 --2- 172.21.15.77:0/1858394878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2690c80 0x5609b26921e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e9137c740 1 --2- 172.21.15.77:0/1858394878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b26aa4c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e9137c740 1 -- 172.21.15.77:0/1858394878 >> 172.21.15.77:0/1858394878 conn(0x5609b26a2720 msgr2=0x5609b26ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:18.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e9137c740 1 -- 172.21.15.77:0/1858394878 shutdown_connections 2024-04-25T01:22:18.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.032+0000 7f9e9137c740 1 -- 172.21.15.77:0/1858394878 wait complete. 2024-04-25T01:22:18.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.033+0000 7f9e9137c740 1 -- start start 2024-04-25T01:22:18.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.033+0000 7f9e9137c740 1 Processor -- start 2024-04-25T01:22:18.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b27285b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2690c80 0x5609b26a8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e9137c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2692c50 0x5609b26a9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e9137c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5609b2691470 con 0x5609b2690540 2024-04-25T01:22:18.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e9137c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5609b26910e0 con 0x5609b2690c80 2024-04-25T01:22:18.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e9137c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5609b2691ac0 con 0x5609b2692c50 2024-04-25T01:22:18.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e9137c740 10 client.0 _ll_register_callbacks cb 0x7f9e78058c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:22:18.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e87fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b27285b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e87fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b27285b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45406/0 (socket says 172.21.15.77:45406) 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e87fff640 1 -- 172.21.15.77:0/2673898297 learned_addr learned my addr 172.21.15.77:0/2673898297 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e877fe640 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2690c80 0x5609b26a8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e8cb2d640 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2692c50 0x5609b26a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e87fff640 1 -- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2692c50 msgr2=0x5609b26a9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e87fff640 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2692c50 0x5609b26a9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e87fff640 1 -- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2690c80 msgr2=0x5609b26a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e87fff640 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2690c80 0x5609b26a8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e87fff640 1 -- 172.21.15.77:0/2673898297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609b26b9fd0 con 0x5609b2690540 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e877fe640 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2690c80 0x5609b26a8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e87fff640 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b27285b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9e78059210 tx=0x7f9e7805c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f372ecda319445 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e66ffd640 1 -- 172.21.15.77:0/2673898297 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e78075070 con 0x5609b2690540 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e66ffd640 1 -- 172.21.15.77:0/2673898297 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9e780588e0 con 0x5609b2690540 2024-04-25T01:22:18.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.034+0000 7f9e9137c740 1 -- 172.21.15.77:0/2673898297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e780891f0 con 0x5609b2690540 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e66ffd640 1 -- 172.21.15.77:0/2673898297 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f9e78082e60 con 0x5609b2690540 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e9137c740 10 client.4836 fetch_fsmap learned FSMap version 9 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e9137c740 1 -- 172.21.15.77:0/2673898297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x5609b26b9fd0 con 0x5609b2690540 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e66ffd640 1 -- 172.21.15.77:0/2673898297 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e780891f0 con 0x5609b2690540 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e66ffd640 1 -- 172.21.15.77:0/2673898297 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f9e78054fd0 con 0x5609b2690540 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e9137c740 10 client.4836 fetch_fsmap finished waiting for FSMap version 9 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e9137c740 10 client.4836 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e9137c740 20 client.4836 populate_metadata read hostname 'smithi077' 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e9137c740 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x5609b26adb60 0x5609b26611e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e9137c740 4 client.4836 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e9137c740 1 -- 172.21.15.77:0/2673898297 --> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] -- command(tid 0: {"prefix": "status"}) -- 0x5609b25d0790 con 0x5609b26adb60 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e877fe640 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x5609b26adb60 0x5609b26611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e877fe640 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x5609b26adb60 0x5609b26611e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77345c9640858986 server_cookie=ee023ba0712c242b in_seq=0 out_seq=0 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.035+0000 7f9e66ffd640 10 client.4836 ms_handle_connect on v2:172.21.15.121:6834/2810316639 2024-04-25T01:22:18.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e66ffd640 1 -- 172.21.15.77:0/2673898297 <== mds.0 v2:172.21.15.121:6834/2810316639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x5609b25d0790 con 0x5609b26adb60 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e66ffd640 10 client.4836 handle_command_reply: tid=0 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 1 client.4836 shutdown 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 1 -- 172.21.15.77:0/2673898297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 msgr2=0x5609b27285b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b27285b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9e78059210 tx=0x7f9e7805c040 comp rx=0 tx=0).stop 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 1 -- 172.21.15.77:0/2673898297 shutdown_connections 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:6834/2810316639,v1:172.21.15.121:6835/2810316639] conn(0x5609b26adb60 0x5609b26611e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5609b2692c50 0x5609b26a9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5609b2690c80 0x5609b26a8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 1 --2- 172.21.15.77:0/2673898297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609b2690540 0x5609b27285b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 1 -- 172.21.15.77:0/2673898297 >> 172.21.15.77:0/2673898297 conn(0x5609b26a2720 msgr2=0x5609b26acb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e66ffd640 0 client.4836 ms_handle_reset on v2:172.21.15.121:6834/2810316639 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 1 -- 172.21.15.77:0/2673898297 shutdown_connections 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 1 -- 172.21.15.77:0/2673898297 wait complete. 2024-04-25T01:22:18.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e9137c740 20 client.4836 trim_cache size 0 max 16384 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "7c3c39ee-077e-476c-8994-eede4fab301d", 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 4294, 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 31.963920300000002, 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 9, 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 24, 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 24, 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 39.206415964999998 2024-04-25T01:22:18.024 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:22:18.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e8e590640 1 -- 172.21.15.77:0/1955787017 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e7406be30 msgr2=0x7f9e7406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e8e590640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e7406be30 0x7f9e7406e2f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9e881f74e0 tx=0x7f9e7c009190 comp rx=0 tx=0).stop 2024-04-25T01:22:18.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e8e590640 1 -- 172.21.15.77:0/1955787017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 msgr2=0x7f9e88159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e8e590640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 0x7f9e88159cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9e78007310 tx=0x7f9e78007710 comp rx=0 tx=0).stop 2024-04-25T01:22:18.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e8e590640 1 -- 172.21.15.77:0/1955787017 shutdown_connections 2024-04-25T01:22:18.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e8e590640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e7406be30 0x7f9e7406e2f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.036+0000 7f9e8e590640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e880ca160 0x7f9e881e6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.037+0000 7f9e8e590640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e8815a4f0 0x7f9e880c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.037+0000 7f9e8e590640 1 --2- 172.21.15.77:0/1955787017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e70004d40 0x7f9e88159cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.037+0000 7f9e8e590640 1 -- 172.21.15.77:0/1955787017 >> 172.21.15.77:0/1955787017 conn(0x7f9e880d3ef0 msgr2=0x7f9e8815c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:18.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.037+0000 7f9e8e590640 1 -- 172.21.15.77:0/1955787017 shutdown_connections 2024-04-25T01:22:18.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.037+0000 7f9e8e590640 1 -- 172.21.15.77:0/1955787017 wait complete. 2024-04-25T01:22:18.028 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T01:22:18.038 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T01:22:18.038 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T01:22:18.039 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T01:22:18.039 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T01:22:18.087 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T01:22:18.087 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-25T01:22:18.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:18.088 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-04-25T01:22:18.103 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-25T01:22:18.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:18.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:22:18.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:18.168 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-04-25T01:22:18.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:18.261 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-04-25T01:22:18.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:18.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-04-25T01:22:18.342 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2024-04-25T01:22:18.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:22:18.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:18.347 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-25T01:22:18.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:18.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2024-04-25T01:22:18.435 INFO:teuthology.orchestra.run.smithi121.stderr:Cannot find device "ceph-brx" 2024-04-25T01:22:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:22:18.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:18.437 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-25T01:22:18.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:18.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2024-04-25T01:22:18.562 INFO:teuthology.orchestra.run.smithi121.stderr:Cannot find device "ceph-brx" 2024-04-25T01:22:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:22:18.563 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-25T01:22:18.563 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:22:18.563 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:22:18.564 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:22:18.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.680+0000 7f28a4e7b640 1 Processor -- start 2024-04-25T01:22:18.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.680+0000 7f28a4e7b640 1 -- start start 2024-04-25T01:22:18.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.681+0000 7f28a4e7b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28a0143940 0x7f28a006e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.681+0000 7f28a4e7b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f28a0143d20 0x7f28a006eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.681+0000 7f28a4e7b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28a0070e60 con 0x7f28a015ecf0 2024-04-25T01:22:18.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.681+0000 7f28a4e7b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f28a0071140 con 0x7f28a0143d20 2024-04-25T01:22:18.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.681+0000 7f28a4e7b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f28a0071420 con 0x7f28a0143940 2024-04-25T01:22:18.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.681+0000 7f289dd9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f28a0143d20 0x7f28a006eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.681+0000 7f289dd9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f28a0143d20 0x7f28a006eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45416/0 (socket says 172.21.15.77:45416) 2024-04-25T01:22:18.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.681+0000 7f289dd9b640 1 -- 172.21.15.77:0/4183206891 learned_addr learned my addr 172.21.15.77:0/4183206891 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:18.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.682+0000 7f289e59c640 1 --2- 172.21.15.77:0/4183206891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28a0143940 0x7f28a006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.682+0000 7f289dd9b640 1 -- 172.21.15.77:0/4183206891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28a0143940 msgr2=0x7f28a006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:18.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.682+0000 7f289dd9b640 1 --2- 172.21.15.77:0/4183206891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28a0143940 0x7f28a006e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.682+0000 7f289dd9b640 1 -- 172.21.15.77:0/4183206891 >> v1:172.21.15.121:6789/0 conn(0x7f28a015ecf0 legacy=0x7f28a015f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.682+0000 7f289dd9b640 1 -- 172.21.15.77:0/4183206891 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a0071700 con 0x7f28a0143d20 2024-04-25T01:22:18.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.682+0000 7f289dd9b640 1 --2- 172.21.15.77:0/4183206891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a0143d20 0x7f28a006eb90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f289400bb10 tx=0x7f289402c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501eeca87c0bfafd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:18.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.682+0000 7f289d59a640 1 -- 172.21.15.77:0/4183206891 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2894036020 con 0x7f28a0143d20 2024-04-25T01:22:18.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.683+0000 7f289d59a640 1 -- 172.21.15.77:0/4183206891 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2894032820 con 0x7f28a0143d20 2024-04-25T01:22:18.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.683+0000 7f289d59a640 1 -- 172.21.15.77:0/4183206891 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2894032bf0 con 0x7f28a0143d20 2024-04-25T01:22:18.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.683+0000 7f28a4e7b640 1 -- 172.21.15.77:0/4183206891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a0143d20 msgr2=0x7f28a006eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.683+0000 7f28a4e7b640 1 --2- 172.21.15.77:0/4183206891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a0143d20 0x7f28a006eb90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f289400bb10 tx=0x7f289402c920 comp rx=0 tx=0).stop 2024-04-25T01:22:18.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.683+0000 7f28a4e7b640 1 -- 172.21.15.77:0/4183206891 shutdown_connections 2024-04-25T01:22:18.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.683+0000 7f28a4e7b640 1 --2- 172.21.15.77:0/4183206891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a0143d20 0x7f28a006eb90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.683+0000 7f28a4e7b640 1 --2- 172.21.15.77:0/4183206891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28a0143940 0x7f28a006e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.683+0000 7f28a4e7b640 1 -- 172.21.15.77:0/4183206891 >> 172.21.15.77:0/4183206891 conn(0x7f28a00b1820 msgr2=0x7f28a00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:18.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.683+0000 7f28a4e7b640 1 -- 172.21.15.77:0/4183206891 shutdown_connections 2024-04-25T01:22:18.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.683+0000 7f28a4e7b640 1 -- 172.21.15.77:0/4183206891 wait complete. 2024-04-25T01:22:18.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.684+0000 7f28a4e7b640 1 Processor -- start 2024-04-25T01:22:18.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.684+0000 7f28a4e7b640 1 -- start start 2024-04-25T01:22:18.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f28a4e7b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a0143940 0x7f28a013eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f28a4e7b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a0143d20 0x7f28a0155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f28a4e7b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28a015ecf0 0x7f28a012ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f28a4e7b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f28a006d8d0 con 0x7f28a0143d20 2024-04-25T01:22:18.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f28a4e7b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f28a006d750 con 0x7f28a0143940 2024-04-25T01:22:18.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f28a4e7b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f28a006da50 con 0x7f28a015ecf0 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f289dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a0143d20 0x7f28a0155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f289dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a0143d20 0x7f28a0155640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45430/0 (socket says 172.21.15.77:45430) 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f289dd9b640 1 -- 172.21.15.77:0/2509019617 learned_addr learned my addr 172.21.15.77:0/2509019617 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f289ed9d640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28a015ecf0 0x7f28a012ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f289e59c640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a0143940 0x7f28a013eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.685+0000 7f289dd9b640 1 -- 172.21.15.77:0/2509019617 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28a015ecf0 msgr2=0x7f28a012ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.686+0000 7f289dd9b640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28a015ecf0 0x7f28a012ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.686+0000 7f289dd9b640 1 -- 172.21.15.77:0/2509019617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a0143940 msgr2=0x7f28a013eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.686+0000 7f289dd9b640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a0143940 0x7f28a013eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.686+0000 7f289dd9b640 1 -- 172.21.15.77:0/2509019617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a0141310 con 0x7f28a0143d20 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.686+0000 7f289ed9d640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28a015ecf0 0x7f28a012ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.686+0000 7f289dd9b640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a0143d20 0x7f28a0155640 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f289402c960 tx=0x7f28940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.686+0000 7f288effd640 1 -- 172.21.15.77:0/2509019617 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2894036040 con 0x7f28a0143d20 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.686+0000 7f288effd640 1 -- 172.21.15.77:0/2509019617 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f289403cd70 con 0x7f28a0143d20 2024-04-25T01:22:18.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.686+0000 7f28a4e7b640 1 -- 172.21.15.77:0/2509019617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28a01415f0 con 0x7f28a0143d20 2024-04-25T01:22:18.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.686+0000 7f288effd640 1 -- 172.21.15.77:0/2509019617 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2894046630 con 0x7f28a0143d20 2024-04-25T01:22:18.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.686+0000 7f28a4e7b640 1 -- 172.21.15.77:0/2509019617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28a0141bb0 con 0x7f28a0143d20 2024-04-25T01:22:18.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.687+0000 7f28a4e7b640 1 -- 172.21.15.77:0/2509019617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28a0068fb0 con 0x7f28a0143d20 2024-04-25T01:22:18.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.687+0000 7f288effd640 1 -- 172.21.15.77:0/2509019617 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f289404d060 con 0x7f28a0143d20 2024-04-25T01:22:18.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.688+0000 7f288effd640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f288006be60 0x7f288006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.688+0000 7f288effd640 1 -- 172.21.15.77:0/2509019617 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f28940ae4e0 con 0x7f28a0143d20 2024-04-25T01:22:18.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.688+0000 7f289e59c640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f288006be60 0x7f288006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.688+0000 7f289e59c640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f288006be60 0x7f288006e320 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2890007330 tx=0x7f289000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:18.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.692+0000 7f288effd640 1 -- 172.21.15.77:0/2509019617 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2894041070 con 0x7f28a0143d20 2024-04-25T01:22:18.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.902+0000 7f28a4e7b640 1 -- 172.21.15.77:0/2509019617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f28a006d150 con 0x7f28a0143d20 2024-04-25T01:22:18.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.902+0000 7f288effd640 1 -- 172.21.15.77:0/2509019617 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f289402f070 con 0x7f28a0143d20 2024-04-25T01:22:18.883 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:22:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 -- 172.21.15.77:0/2509019617 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f288006be60 msgr2=0x7f288006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f288006be60 0x7f288006e320 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2890007330 tx=0x7f289000b040 comp rx=0 tx=0).stop 2024-04-25T01:22:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 -- 172.21.15.77:0/2509019617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a0143d20 msgr2=0x7f28a0155640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a0143d20 0x7f28a0155640 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f289402c960 tx=0x7f28940046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 -- 172.21.15.77:0/2509019617 shutdown_connections 2024-04-25T01:22:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f288006be60 0x7f288006e320 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a0143d20 0x7f28a0155640 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28a015ecf0 0x7f28a012ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 --2- 172.21.15.77:0/2509019617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a0143940 0x7f28a013eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 -- 172.21.15.77:0/2509019617 >> 172.21.15.77:0/2509019617 conn(0x7f28a00b1820 msgr2=0x7f28a00b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 -- 172.21.15.77:0/2509019617 shutdown_connections 2024-04-25T01:22:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:18.904+0000 7f28a4e7b640 1 -- 172.21.15.77:0/2509019617 wait complete. 2024-04-25T01:22:18.896 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:22:18.896 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:22:18.896 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-04-25T01:22:18.896 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:22:18.896 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:22:18.896 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:22:18.896 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:22:18.896 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:22:18.896 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:22:18.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:18.897 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-04-25T01:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-04-25T01:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42710sec preferred_lft 42710sec 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a9ad:d2f:d456:e65f/64 scope link noprefixroute 2024-04-25T01:22:18.913 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:18.914 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:22:18.914 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:18.914 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:22:18.914 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:22:18.914 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr flush dev ceph-brx 2024-04-25T01:22:18.914 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set ceph-brx up 2024-04-25T01:22:18.914 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:22:18.914 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:22:19.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:19.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:19.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:19.046 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:22:19.068 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-04-25T01:22:19.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:19.069 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-04-25T01:22:19.125 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-04-25T01:22:19.125 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:22:19.125 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:22:19.125 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:22:19.125 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:22:19.126 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:19.126 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:22:19.126 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:22:19.126 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:22:19.126 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:22:19.126 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:22:19.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:19.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:19.249 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-04-25T01:22:19.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:19.263 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-04-25T01:22:19.318 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:19.318 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:22:19.318 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:22:19.318 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T01:22:19.318 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:22:19.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:19.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:19.433 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T01:22:19.433 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:19.433 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:22:19.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T01:22:19.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:22:19.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T01:22:19.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T01:22:19.434 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T01:22:19.434 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:22:19.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:19.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:19.540 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:19.540 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:22:19.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-04-25T01:22:19.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:22:19.541 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:22:19.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:19.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:19.615 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:22:19.615 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T01:22:19.629 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:22:19.629 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:22:19.629 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:22:19.684 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:22:19.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:19.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:22:19.766 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:22:19.767 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:22:19.767 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:22:19.878 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:22:19.878 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:22:19.878 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:22:19.878 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:22:19.878 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:22:19.883 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:22:19.883 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:22:19.883 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:22:19.883 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:22:19.883 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:22:19.883 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790" 2024-04-25T01:22:19.883 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790:/ 2024-04-25T01:22:19.883 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=7c3c39ee-077e-476c-8994-eede4fab301d" will pass to kernel 2024-04-25T01:22:19.884 INFO:teuthology.orchestra.run.smithi077.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T01:22:19.884 INFO:teuthology.orchestra.run.smithi077.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:22:19.884 INFO:teuthology.orchestra.run.smithi077.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:22:19.884 INFO:teuthology.orchestra.run.smithi077.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:22:19.884 INFO:teuthology.orchestra.run.smithi077.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:22:19.886 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:22:19.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:19.886 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:22:19.951 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:20.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:20.047 INFO:teuthology.orchestra.run.smithi077.stdout:client4693 2024-04-25T01:22:20.047 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:22:20.047 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:20.062 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:20.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:20.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:20.157 INFO:teuthology.orchestra.run.smithi077.stdout:client4693 2024-04-25T01:22:20.157 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:20.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:20.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:20.212 INFO:teuthology.orchestra.run.smithi077.stdout:client4693 2024-04-25T01:22:20.212 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/7c3c39ee-077e-476c-8994-eede4fab301d.client4693/status') 2024-04-25T01:22:20.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:20.269 INFO:teuthology.orchestra.run.smithi077.stdout:instance: client.4693 (3)172.21.15.77:0/2828147224 2024-04-25T01:22:20.269 INFO:teuthology.orchestra.run.smithi077.stdout:blocklisted: false 2024-04-25T01:22:20.270 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records in 2024-04-25T01:22:20.270 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records out 2024-04-25T01:22:20.270 INFO:teuthology.orchestra.run.smithi077.stderr:70 bytes copied, 2.1334e-05 s, 3.3 MB/s 2024-04-25T01:22:20.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:20.277 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-04-25T01:22:20.277 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:22:20.277 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T01:22:20.277 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:22:20.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.392+0000 7fc7b2b68640 1 Processor -- start 2024-04-25T01:22:20.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.392+0000 7fc7b2b68640 1 -- start start 2024-04-25T01:22:20.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.393+0000 7fc7b2b68640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc7ac0d4a10 0x7fc7ac0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:20.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.393+0000 7fc7b2b68640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac0b3ee0 0x7fc7ac0b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:20.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.393+0000 7fc7b2b68640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ac0d1e90 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.393+0000 7fc7b2b68640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fc7ac0b54a0 con 0x7fc7ac0d4a10 2024-04-25T01:22:20.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.393+0000 7fc7b2b68640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc7ac0d2130 con 0x7fc7ac0b3ee0 2024-04-25T01:22:20.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.393+0000 7fc7abfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc7ac0d4a10 0x7fc7ac0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:20.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.393+0000 7fc7b0904640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc7ac15e4c0 0x7fc7ac0d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36626/0 (socket says 172.21.15.77:36626) 2024-04-25T01:22:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.393+0000 7fc7b0904640 1 -- 172.21.15.77:0/3630140029 learned_addr learned my addr 172.21.15.77:0/3630140029 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.394+0000 7fc7b1105640 1 --2- 172.21.15.77:0/3630140029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac0b3ee0 0x7fc7ac0b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.394+0000 7fc7abfff640 1 -- 172.21.15.77:0/3630140029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac0b3ee0 msgr2=0x7fc7ac0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.394+0000 7fc7abfff640 1 --2- 172.21.15.77:0/3630140029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac0b3ee0 0x7fc7ac0b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.394+0000 7fc7abfff640 1 -- 172.21.15.77:0/3630140029 >> v1:172.21.15.77:6789/0 conn(0x7fc7ac15e4c0 legacy=0x7fc7ac0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.394+0000 7fc7abfff640 1 -- 172.21.15.77:0/3630140029 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ac0b5780 con 0x7fc7ac0d4a10 2024-04-25T01:22:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.394+0000 7fc7ab7fe640 1 -- 172.21.15.77:0/3630140029 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3861487285 0 0) 0x7fc7ac0d1e90 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.394+0000 7fc7abfff640 1 --2- 172.21.15.77:0/3630140029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ac0d4a10 0x7fc7ac0cfaf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00097a0 tx=0x7fc7a002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c23109a73a43773 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.394+0000 7fc7ab7fe640 1 -- 172.21.15.77:0/3630140029 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0037070 con 0x7fc7ac0d4a10 2024-04-25T01:22:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.394+0000 7fc7ab7fe640 1 -- 172.21.15.77:0/3630140029 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7a002ae50 con 0x7fc7ac0d4a10 2024-04-25T01:22:20.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.394+0000 7fc7ab7fe640 1 -- 172.21.15.77:0/3630140029 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0032660 con 0x7fc7ac0d4a10 2024-04-25T01:22:20.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.395+0000 7fc7b2b68640 1 -- 172.21.15.77:0/3630140029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ac0d4a10 msgr2=0x7fc7ac0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:20.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.395+0000 7fc7b2b68640 1 --2- 172.21.15.77:0/3630140029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ac0d4a10 0x7fc7ac0cfaf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00097a0 tx=0x7fc7a002a100 comp rx=0 tx=0).stop 2024-04-25T01:22:20.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.395+0000 7fc7b2b68640 1 -- 172.21.15.77:0/3630140029 shutdown_connections 2024-04-25T01:22:20.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.395+0000 7fc7b2b68640 1 --2- 172.21.15.77:0/3630140029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac0b3ee0 0x7fc7ac0b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.395+0000 7fc7b2b68640 1 --2- 172.21.15.77:0/3630140029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ac0d4a10 0x7fc7ac0cfaf0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.395+0000 7fc7b2b68640 1 -- 172.21.15.77:0/3630140029 >> 172.21.15.77:0/3630140029 conn(0x7fc7ac0d3ef0 msgr2=0x7fc7ac0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:20.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.395+0000 7fc7b2b68640 1 -- 172.21.15.77:0/3630140029 shutdown_connections 2024-04-25T01:22:20.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.395+0000 7fc7b2b68640 1 -- 172.21.15.77:0/3630140029 wait complete. 2024-04-25T01:22:20.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.396+0000 7fc7b2b68640 1 Processor -- start 2024-04-25T01:22:20.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.396+0000 7fc7b2b68640 1 -- start start 2024-04-25T01:22:20.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b2b68640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7ac0b3ee0 0x7fc7ac0b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:20.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b2b68640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac0d4a10 0x7fc7ac151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:20.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b2b68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ac15e4c0 0x7fc7ac0c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:20.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b2b68640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac0b30a0 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b2b68640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac0b2f20 con 0x7fc7ac0b3ee0 2024-04-25T01:22:20.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b2b68640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc7ac15e8a0 con 0x7fc7ac0d4a10 2024-04-25T01:22:20.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b1105640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ac15e4c0 0x7fc7ac0c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:20.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b1105640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ac15e4c0 0x7fc7ac0c9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45446/0 (socket says 172.21.15.77:45446) 2024-04-25T01:22:20.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b1105640 1 -- 172.21.15.77:0/2059042320 learned_addr learned my addr 172.21.15.77:0/2059042320 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:20.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7abfff640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac0d4a10 0x7fc7ac151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:20.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b0904640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7ac0b3ee0 0x7fc7ac0b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:20.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b1105640 1 -- 172.21.15.77:0/2059042320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac0d4a10 msgr2=0x7fc7ac151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:20.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b1105640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac0d4a10 0x7fc7ac151340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b1105640 1 -- 172.21.15.77:0/2059042320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7ac0b3ee0 msgr2=0x7fc7ac0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:20.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b1105640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7ac0b3ee0 0x7fc7ac0b4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.397+0000 7fc7b1105640 1 -- 172.21.15.77:0/2059042320 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ac151880 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.398+0000 7fc7b1105640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ac15e4c0 0x7fc7ac0c9bf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4007ea0 tx=0x7fc7a4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:20.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.398+0000 7fc7a97fa640 1 -- 172.21.15.77:0/2059042320 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4014620 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.398+0000 7fc7a97fa640 1 -- 172.21.15.77:0/2059042320 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7a4019070 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.398+0000 7fc7b2b68640 1 -- 172.21.15.77:0/2059042320 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7ac0b9530 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.398+0000 7fc7a97fa640 1 -- 172.21.15.77:0/2059042320 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4014a20 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.398+0000 7fc7b2b68640 1 -- 172.21.15.77:0/2059042320 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7ac0b9af0 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.399+0000 7fc7b2b68640 1 -- 172.21.15.77:0/2059042320 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7ac068fb0 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.399+0000 7fc7a97fa640 1 -- 172.21.15.77:0/2059042320 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc7a4014bc0 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.399+0000 7fc7a97fa640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc79806be60 0x7fc79806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:20.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.400+0000 7fc7a97fa640 1 -- 172.21.15.77:0/2059042320 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc7a408ba80 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.400+0000 7fc7b0904640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc79806be60 0x7fc79806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:20.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.400+0000 7fc7b0904640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc79806be60 0x7fc79806e320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc79c000f60 tx=0x7fc79c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:20.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.404+0000 7fc7a97fa640 1 -- 172.21.15.77:0/2059042320 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc7a405a060 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.613+0000 7fc7b2b68640 1 -- 172.21.15.77:0/2059042320 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc7ac073c20 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.613+0000 7fc7a97fa640 1 -- 172.21.15.77:0/2059042320 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fc7a4059960 con 0x7fc7ac15e4c0 2024-04-25T01:22:20.594 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:22:20.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.615+0000 7fc7b2b68640 1 -- 172.21.15.77:0/2059042320 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc79806be60 msgr2=0x7fc79806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.615+0000 7fc7b2b68640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc79806be60 0x7fc79806e320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc79c000f60 tx=0x7fc79c009040 comp rx=0 tx=0).stop 2024-04-25T01:22:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.615+0000 7fc7b2b68640 1 -- 172.21.15.77:0/2059042320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ac15e4c0 msgr2=0x7fc7ac0c9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.615+0000 7fc7b2b68640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ac15e4c0 0x7fc7ac0c9bf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4007ea0 tx=0x7fc7a4007f90 comp rx=0 tx=0).stop 2024-04-25T01:22:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.615+0000 7fc7b2b68640 1 -- 172.21.15.77:0/2059042320 shutdown_connections 2024-04-25T01:22:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.615+0000 7fc7b2b68640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc79806be60 0x7fc79806e320 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.615+0000 7fc7b2b68640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7ac15e4c0 0x7fc7ac0c9bf0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.615+0000 7fc7b2b68640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac0d4a10 0x7fc7ac151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.615+0000 7fc7b2b68640 1 --2- 172.21.15.77:0/2059042320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7ac0b3ee0 0x7fc7ac0b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.615+0000 7fc7b2b68640 1 -- 172.21.15.77:0/2059042320 >> 172.21.15.77:0/2059042320 conn(0x7fc7ac0d3ef0 msgr2=0x7fc7ac0d2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.615+0000 7fc7b2b68640 1 -- 172.21.15.77:0/2059042320 shutdown_connections 2024-04-25T01:22:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20.616+0000 7fc7b2b68640 1 -- 172.21.15.77:0/2059042320 wait complete. 2024-04-25T01:22:20.607 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:22:20.607 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:22:20.607 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-04-25T01:22:20.607 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-25T01:22:20.607 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T01:22:20.607 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:22:20.607 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:22:20.607 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:22:20.607 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-25T01:22:20.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:20.607 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42708sec preferred_lft 42708sec 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a9ad:d2f:d456:e65f/64 scope link noprefixroute 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0a:90:ec:3f:d1:57 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::b40c:78ff:fecf:cd0c/64 scope link 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi077.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0a:90:ec:3f:d1:57 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::890:ecff:fe3f:d157/64 scope link tentative 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:20.625 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:22:20.685 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-04-25T01:22:20.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:20.687 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-04-25T01:22:20.743 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-04-25T01:22:20.743 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:22:20.743 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:22:20.743 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:22:20.743 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:22:20.744 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:20.744 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:22:20.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:22:20.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:22:20.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:22:20.744 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:22:20.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:20.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:20.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:20.866 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-04-25T01:22:20.881 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:22:20.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:20.881 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-04-25T01:22:20.937 INFO:teuthology.orchestra.run.smithi077.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-25T01:22:20.937 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:20.937 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:22:20.937 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T01:22:20.937 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-25T01:22:20.937 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:22:21.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:21.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:21.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:21.051 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T01:22:21.074 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:22:21.075 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:22:21.075 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:22:21.075 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:21.075 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-04-25T01:22:21.075 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:21.075 INFO:teuthology.orchestra.run.smithi077.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:22:21.075 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 8e:a0:ec:ec:bc:3b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T01:22:21.075 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T01:22:21.075 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:21.075 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::8ca0:ecff:feec:bc3b/64 scope link tentative 2024-04-25T01:22:21.075 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:21.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:21.076 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T01:22:21.139 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:22:21.139 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:22:21.139 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:22:21.140 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:21.140 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-04-25T01:22:21.140 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:21.140 INFO:teuthology.orchestra.run.smithi077.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:22:21.140 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 8e:a0:ec:ec:bc:3b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T01:22:21.140 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T01:22:21.140 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:21.140 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::8ca0:ecff:feec:bc3b/64 scope link 2024-04-25T01:22:21.140 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:21.141 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-25T01:22:21.141 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:21.141 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:22:21.141 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-25T01:22:21.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T01:22:21.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-25T01:22:21.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-25T01:22:21.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-25T01:22:21.142 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:22:21.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:21.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:21.288 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:21.288 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:22:21.288 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.1 up 2024-04-25T01:22:21.288 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T01:22:21.288 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:22:21.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:21.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:21.362 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:22:21.362 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-25T01:22:21.376 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-25T01:22:21.377 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:22:21.377 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-25T01:22:21.431 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:22:21.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:21.431 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-25T01:22:21.512 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-25T01:22:21.513 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-25T01:22:21.513 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:22:21.521 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-25T01:22:21.521 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-25T01:22:21.521 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:22:21.521 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:22:21.521 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:22:21.526 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-25T01:22:21.526 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-25T01:22:21.526 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:22:21.526 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:22:21.526 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:22:21.526 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790" 2024-04-25T01:22:21.526 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790:/ 2024-04-25T01:22:21.526 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=7c3c39ee-077e-476c-8994-eede4fab301d" will pass to kernel 2024-04-25T01:22:21.527 INFO:teuthology.orchestra.run.smithi077.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-04-25T01:22:21.527 INFO:teuthology.orchestra.run.smithi077.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:22:21.527 INFO:teuthology.orchestra.run.smithi077.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:22:21.527 INFO:teuthology.orchestra.run.smithi077.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:22:21.527 INFO:teuthology.orchestra.run.smithi077.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:22:21.528 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:22:21.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:21.528 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-25T01:22:21.592 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:21.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:21.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:21.688 INFO:teuthology.orchestra.run.smithi077.stdout:client4559 2024-04-25T01:22:21.689 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:22:21.689 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:21.703 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:21.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:21.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:21.798 INFO:teuthology.orchestra.run.smithi077.stdout:client4559 2024-04-25T01:22:21.799 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:21.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:21.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:21.852 INFO:teuthology.orchestra.run.smithi077.stdout:client4559 2024-04-25T01:22:21.853 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/7c3c39ee-077e-476c-8994-eede4fab301d.client4559/status') 2024-04-25T01:22:21.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:21.909 INFO:teuthology.orchestra.run.smithi077.stdout:instance: client.4559 (3)172.21.15.77:0/3212205991 2024-04-25T01:22:21.909 INFO:teuthology.orchestra.run.smithi077.stdout:blocklisted: false 2024-04-25T01:22:21.909 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records in 2024-04-25T01:22:21.909 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records out 2024-04-25T01:22:21.909 INFO:teuthology.orchestra.run.smithi077.stderr:70 bytes copied, 2.1727e-05 s, 3.2 MB/s 2024-04-25T01:22:21.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:21.916 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-04-25T01:22:21.916 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:22:21.917 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T01:22:21.917 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:22:22.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.032+0000 7f342334c640 1 Processor -- start 2024-04-25T01:22:22.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.032+0000 7f342334c640 1 -- start start 2024-04-25T01:22:22.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.033+0000 7f342334c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f341c143d20 0x7f341c15ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:22.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.033+0000 7f342334c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f341c06ed60 0x7f341c15f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:22.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.033+0000 7f342334c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f341c15f770 con 0x7f341c143940 2024-04-25T01:22:22.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.033+0000 7f342334c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f341c070e60 con 0x7f341c06ed60 2024-04-25T01:22:22.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.033+0000 7f342334c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f341c071140 con 0x7f341c143d20 2024-04-25T01:22:22.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.033+0000 7f34218e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f341c06ed60 0x7f341c15f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:22.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.033+0000 7f34218e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f341c06ed60 0x7f341c15f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45460/0 (socket says 172.21.15.77:45460) 2024-04-25T01:22:22.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.033+0000 7f34218e9640 1 -- 172.21.15.77:0/817488403 learned_addr learned my addr 172.21.15.77:0/817488403 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:22.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.033+0000 7f34208e7640 1 --2- 172.21.15.77:0/817488403 >> v2:172.21.15.121:3300/0 conn(0x7f341c143d20 0x7f341c15ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:22.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.034+0000 7f34218e9640 1 -- 172.21.15.77:0/817488403 >> v2:172.21.15.121:3300/0 conn(0x7f341c143d20 msgr2=0x7f341c15ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:22.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.034+0000 7f34218e9640 1 --2- 172.21.15.77:0/817488403 >> v2:172.21.15.121:3300/0 conn(0x7f341c143d20 0x7f341c15ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.034+0000 7f34218e9640 1 -- 172.21.15.77:0/817488403 >> v1:172.21.15.121:6789/0 conn(0x7f341c143940 legacy=0x7f341c06e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:22.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.034+0000 7f34218e9640 1 -- 172.21.15.77:0/817488403 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c071420 con 0x7f341c06ed60 2024-04-25T01:22:22.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.034+0000 7f34218e9640 1 --2- 172.21.15.77:0/817488403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c06ed60 0x7f341c15f230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3414009e50 tx=0x7f341402a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb8f92188f6d82f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:22.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.034+0000 7f3413fff640 1 -- 172.21.15.77:0/817488403 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414036070 con 0x7f341c06ed60 2024-04-25T01:22:22.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.034+0000 7f3413fff640 1 -- 172.21.15.77:0/817488403 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3414031600 con 0x7f341c06ed60 2024-04-25T01:22:22.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.034+0000 7f3413fff640 1 -- 172.21.15.77:0/817488403 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34140319d0 con 0x7f341c06ed60 2024-04-25T01:22:22.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.035+0000 7f342334c640 1 -- 172.21.15.77:0/817488403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c06ed60 msgr2=0x7f341c15f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:22.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.035+0000 7f342334c640 1 --2- 172.21.15.77:0/817488403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c06ed60 0x7f341c15f230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3414009e50 tx=0x7f341402a760 comp rx=0 tx=0).stop 2024-04-25T01:22:22.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.035+0000 7f342334c640 1 -- 172.21.15.77:0/817488403 shutdown_connections 2024-04-25T01:22:22.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.035+0000 7f342334c640 1 --2- 172.21.15.77:0/817488403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c06ed60 0x7f341c15f230 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.035+0000 7f342334c640 1 --2- 172.21.15.77:0/817488403 >> v2:172.21.15.121:3300/0 conn(0x7f341c143d20 0x7f341c15ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.035+0000 7f342334c640 1 -- 172.21.15.77:0/817488403 >> 172.21.15.77:0/817488403 conn(0x7f341c0b1820 msgr2=0x7f341c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:22.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.035+0000 7f342334c640 1 -- 172.21.15.77:0/817488403 shutdown_connections 2024-04-25T01:22:22.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.035+0000 7f342334c640 1 -- 172.21.15.77:0/817488403 wait complete. 2024-04-25T01:22:22.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.036+0000 7f342334c640 1 Processor -- start 2024-04-25T01:22:22.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.036+0000 7f342334c640 1 -- start start 2024-04-25T01:22:22.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f342334c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f341c06ed60 0x7f341c153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:22.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f342334c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c143940 0x7f341c142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:22.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f342334c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341c143d20 0x7f341c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:22.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f342334c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f341c071da0 con 0x7f341c143940 2024-04-25T01:22:22.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f342334c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f341c071c20 con 0x7f341c06ed60 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f342334c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f341c071f20 con 0x7f341c143d20 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34208e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c143940 0x7f341c142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34208e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c143940 0x7f341c142840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45476/0 (socket says 172.21.15.77:45476) 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34208e7640 1 -- 172.21.15.77:0/1349280138 learned_addr learned my addr 172.21.15.77:0/1349280138 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34210e8640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f341c06ed60 0x7f341c153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34218e9640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341c143d20 0x7f341c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34208e7640 1 -- 172.21.15.77:0/1349280138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341c143d20 msgr2=0x7f341c06d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34208e7640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341c143d20 0x7f341c06d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34208e7640 1 -- 172.21.15.77:0/1349280138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f341c06ed60 msgr2=0x7f341c153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34208e7640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f341c06ed60 0x7f341c153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34208e7640 1 -- 172.21.15.77:0/1349280138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c142d80 con 0x7f341c143940 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34210e8640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f341c06ed60 0x7f341c153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.037+0000 7f34208e7640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c143940 0x7f341c142840 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f340c00d600 tx=0x7f340c00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.038+0000 7f3411ffb640 1 -- 172.21.15.77:0/1349280138 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c0125f0 con 0x7f341c143940 2024-04-25T01:22:22.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.038+0000 7f3411ffb640 1 -- 172.21.15.77:0/1349280138 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f340c019070 con 0x7f341c143940 2024-04-25T01:22:22.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.038+0000 7f342334c640 1 -- 172.21.15.77:0/1349280138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f341c06e120 con 0x7f341c143940 2024-04-25T01:22:22.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.038+0000 7f3411ffb640 1 -- 172.21.15.77:0/1349280138 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c0129f0 con 0x7f341c143940 2024-04-25T01:22:22.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.038+0000 7f342334c640 1 -- 172.21.15.77:0/1349280138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f341c06de40 con 0x7f341c143940 2024-04-25T01:22:22.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.039+0000 7f342334c640 1 -- 172.21.15.77:0/1349280138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f341c0b34f0 con 0x7f341c143940 2024-04-25T01:22:22.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.039+0000 7f3411ffb640 1 -- 172.21.15.77:0/1349280138 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f340c012b90 con 0x7f341c143940 2024-04-25T01:22:22.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.040+0000 7f3411ffb640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f340006be30 0x7f340006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:22.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.040+0000 7f3411ffb640 1 -- 172.21.15.77:0/1349280138 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f340c08b4f0 con 0x7f341c143940 2024-04-25T01:22:22.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.040+0000 7f34210e8640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f340006be30 0x7f340006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:22.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.040+0000 7f34210e8640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f340006be30 0x7f340006e2f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3418001700 tx=0x7f3418009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:22.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.044+0000 7f3411ffb640 1 -- 172.21.15.77:0/1349280138 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f341c0b34f0 con 0x7f341c143940 2024-04-25T01:22:22.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.260+0000 7f342334c640 1 -- 172.21.15.77:0/1349280138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f341c068fb0 con 0x7f341c143940 2024-04-25T01:22:22.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.261+0000 7f3411ffb640 1 -- 172.21.15.77:0/1349280138 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f340c059bf0 con 0x7f341c143940 2024-04-25T01:22:22.241 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:22:22.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 -- 172.21.15.77:0/1349280138 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f340006be30 msgr2=0x7f340006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:22.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f340006be30 0x7f340006e2f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3418001700 tx=0x7f3418009040 comp rx=0 tx=0).stop 2024-04-25T01:22:22.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 -- 172.21.15.77:0/1349280138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c143940 msgr2=0x7f341c142840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:22.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c143940 0x7f341c142840 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f340c00d600 tx=0x7f340c00dac0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 -- 172.21.15.77:0/1349280138 shutdown_connections 2024-04-25T01:22:22.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f340006be30 0x7f340006e2f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c143940 0x7f341c142840 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341c143d20 0x7f341c06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 --2- 172.21.15.77:0/1349280138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f341c06ed60 0x7f341c153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 -- 172.21.15.77:0/1349280138 >> 172.21.15.77:0/1349280138 conn(0x7f341c0b1820 msgr2=0x7f341c0ce980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:22.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 -- 172.21.15.77:0/1349280138 shutdown_connections 2024-04-25T01:22:22.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:22.263+0000 7f342334c640 1 -- 172.21.15.77:0/1349280138 wait complete. 2024-04-25T01:22:22.254 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:22:22.254 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:22:22.254 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi121.front.sepia.ceph.com 2024-04-25T01:22:22.254 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T01:22:22.254 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T01:22:22.254 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:22:22.255 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:22:22.255 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:22:22.255 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T01:22:22.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:22.255 DEBUG:teuthology.orchestra.run.smithi121:> ip addr 2024-04-25T01:22:22.274 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:22:22.274 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:22:22.274 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:22:22.274 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:22.274 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2024-04-25T01:22:22.274 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:22.274 INFO:teuthology.orchestra.run.smithi121.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:22.274 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:82 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:22.274 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f0 2024-04-25T01:22:22.274 INFO:teuthology.orchestra.run.smithi121.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:22.274 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:83 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:22.275 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f1 2024-04-25T01:22:22.275 INFO:teuthology.orchestra.run.smithi121.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:22.275 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a2 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:22.275 INFO:teuthology.orchestra.run.smithi121.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:22:22.275 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a3 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:22.275 INFO:teuthology.orchestra.run.smithi121.stdout: inet 172.21.15.121/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T01:22:22.275 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft 42708sec preferred_lft 42708sec 2024-04-25T01:22:22.275 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::5365:1d8c:566a:b419/64 scope link noprefixroute 2024-04-25T01:22:22.275 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:22.275 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:22:22.275 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:22.276 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-25T01:22:22.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:22:22.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr flush dev ceph-brx 2024-04-25T01:22:22.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set ceph-brx up 2024-04-25T01:22:22.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:22:22.276 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-25T01:22:22.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:22.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:22.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:22.667 DEBUG:teuthology.orchestra.run.smithi121:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:22:22.691 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-04-25T01:22:22.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:22.693 DEBUG:teuthology.orchestra.run.smithi121:> route 2024-04-25T01:22:22.760 INFO:teuthology.orchestra.run.smithi121.stdout:Kernel IP routing table 2024-04-25T01:22:22.760 INFO:teuthology.orchestra.run.smithi121.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:22:22.760 INFO:teuthology.orchestra.run.smithi121.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:22:22.760 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:22:22.760 INFO:teuthology.orchestra.run.smithi121.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:22:22.760 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:22.761 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-25T01:22:22.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:22:22.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:22:22.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:22:22.761 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-25T01:22:22.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:23.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:23.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:23.040 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-25T01:22:23.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:23.055 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list-id 2024-04-25T01:22:23.110 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:23.110 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-25T01:22:23.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T01:22:23.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-25T01:22:23.110 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-25T01:22:23.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:23.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:23.225 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-25T01:22:23.226 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:23.226 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-25T01:22:23.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-25T01:22:23.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:22:23.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-25T01:22:23.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-25T01:22:23.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-25T01:22:23.226 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-25T01:22:23.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:23.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:23.345 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:23.345 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-25T01:22:23.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set brx.0 up 2024-04-25T01:22:23.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:22:23.345 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-25T01:22:23.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:23.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:23.419 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:22:23.419 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T01:22:23.433 INFO:teuthology.orchestra.run.smithi121.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T01:22:23.433 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:22:23.433 DEBUG:teuthology.orchestra.run.smithi121:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T01:22:23.487 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:22:23.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:23.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-25T01:22:23.580 INFO:teuthology.orchestra.run.smithi121.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-25T01:22:23.580 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-25T01:22:23.580 INFO:teuthology.orchestra.run.smithi121.stdout:invalid new device string format 2024-04-25T01:22:23.689 INFO:teuthology.orchestra.run.smithi121.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-25T01:22:23.690 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-25T01:22:23.690 INFO:teuthology.orchestra.run.smithi121.stdout:invalid new device string format 2024-04-25T01:22:23.690 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:22:23.690 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:22:23.694 INFO:teuthology.orchestra.run.smithi121.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-25T01:22:23.694 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-25T01:22:23.695 INFO:teuthology.orchestra.run.smithi121.stdout:invalid new device string format 2024-04-25T01:22:23.695 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:22:23.695 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:22:23.695 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790" 2024-04-25T01:22:23.695 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790:/ 2024-04-25T01:22:23.695 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=7c3c39ee-077e-476c-8994-eede4fab301d" will pass to kernel 2024-04-25T01:22:23.697 INFO:teuthology.orchestra.run.smithi121.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T01:22:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:22:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:22:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:22:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:22:23.698 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:22:23.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:23.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T01:22:23.721 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:23.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:23.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:23.817 INFO:teuthology.orchestra.run.smithi121.stdout:client4565 2024-04-25T01:22:23.818 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:22:23.818 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:23.833 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:23.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:23.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:23.928 INFO:teuthology.orchestra.run.smithi121.stdout:client4565 2024-04-25T01:22:23.928 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:23.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:23.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:23.983 INFO:teuthology.orchestra.run.smithi121.stdout:client4565 2024-04-25T01:22:23.983 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/7c3c39ee-077e-476c-8994-eede4fab301d.client4565/status') 2024-04-25T01:22:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:24.040 INFO:teuthology.orchestra.run.smithi121.stdout:instance: client.4565 (3)192.168.0.1:0/1722750552 2024-04-25T01:22:24.040 INFO:teuthology.orchestra.run.smithi121.stdout:blocklisted: false 2024-04-25T01:22:24.040 INFO:teuthology.orchestra.run.smithi121.stderr:0+1 records in 2024-04-25T01:22:24.040 INFO:teuthology.orchestra.run.smithi121.stderr:0+1 records out 2024-04-25T01:22:24.040 INFO:teuthology.orchestra.run.smithi121.stderr:69 bytes copied, 2.175e-05 s, 3.2 MB/s 2024-04-25T01:22:24.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:24.047 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-04-25T01:22:24.048 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:22:24.048 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T01:22:24.048 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:22:24.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.176+0000 7f6ca36c1640 1 Processor -- start 2024-04-25T01:22:24.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.177+0000 7f6ca36c1640 1 -- start start 2024-04-25T01:22:24.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.177+0000 7f6ca36c1640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6c9c0d4a10 0x7f6c9c0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.177+0000 7f6ca36c1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c9c0b3ee0 0x7f6c9c0b4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.177+0000 7f6ca36c1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c9c0d1e90 con 0x7f6c9c15e4c0 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.177+0000 7f6ca36c1640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6c9c0b54a0 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.177+0000 7f6ca36c1640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6c9c0d2130 con 0x7f6c9c0d4a10 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.178+0000 7f6ca1c5e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c9c0b3ee0 0x7f6c9c0b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.178+0000 7f6ca1c5e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c9c0b3ee0 0x7f6c9c0b4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45478/0 (socket says 172.21.15.77:45478) 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.178+0000 7f6ca1c5e640 1 -- 172.21.15.77:0/2111434088 learned_addr learned my addr 172.21.15.77:0/2111434088 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.178+0000 7f6ca0c5c640 1 --2- 172.21.15.77:0/2111434088 >> v2:172.21.15.121:3300/0 conn(0x7f6c9c0d4a10 0x7f6c9c0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.178+0000 7f6ca1c5e640 1 -- 172.21.15.77:0/2111434088 >> v2:172.21.15.121:3300/0 conn(0x7f6c9c0d4a10 msgr2=0x7f6c9c0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.178+0000 7f6ca1c5e640 1 --2- 172.21.15.77:0/2111434088 >> v2:172.21.15.121:3300/0 conn(0x7f6c9c0d4a10 0x7f6c9c0cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.178+0000 7f6ca1c5e640 1 -- 172.21.15.77:0/2111434088 >> v1:172.21.15.121:6789/0 conn(0x7f6c9c15e4c0 legacy=0x7f6c9c0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:24.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.178+0000 7f6ca1c5e640 1 -- 172.21.15.77:0/2111434088 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9c0b5780 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6ca1c5e640 1 --2- 172.21.15.77:0/2111434088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c9c0b3ee0 0x7f6c9c0b4370 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c0d4ff0 tx=0x7f6c9402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75894f3e3d3fa460 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:24.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6c93fff640 1 -- 172.21.15.77:0/2111434088 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c94036070 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6c93fff640 1 -- 172.21.15.77:0/2111434088 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c94031600 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6c93fff640 1 -- 172.21.15.77:0/2111434088 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c940319d0 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6ca36c1640 1 -- 172.21.15.77:0/2111434088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c9c0b3ee0 msgr2=0x7f6c9c0b4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:24.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6ca36c1640 1 --2- 172.21.15.77:0/2111434088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c9c0b3ee0 0x7f6c9c0b4370 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c0d4ff0 tx=0x7f6c9402a6a0 comp rx=0 tx=0).stop 2024-04-25T01:22:24.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6ca36c1640 1 -- 172.21.15.77:0/2111434088 shutdown_connections 2024-04-25T01:22:24.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6ca36c1640 1 --2- 172.21.15.77:0/2111434088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c9c0b3ee0 0x7f6c9c0b4370 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:24.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6ca36c1640 1 --2- 172.21.15.77:0/2111434088 >> v2:172.21.15.121:3300/0 conn(0x7f6c9c0d4a10 0x7f6c9c0cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:24.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6ca36c1640 1 -- 172.21.15.77:0/2111434088 >> 172.21.15.77:0/2111434088 conn(0x7f6c9c0d3ef0 msgr2=0x7f6c9c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:24.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6ca36c1640 1 -- 172.21.15.77:0/2111434088 shutdown_connections 2024-04-25T01:22:24.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.179+0000 7f6ca36c1640 1 -- 172.21.15.77:0/2111434088 wait complete. 2024-04-25T01:22:24.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.181+0000 7f6ca36c1640 1 Processor -- start 2024-04-25T01:22:24.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.181+0000 7f6ca36c1640 1 -- start start 2024-04-25T01:22:24.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.181+0000 7f6ca36c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c9c0b3ee0 0x7f6c9c0b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:24.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.181+0000 7f6ca36c1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c9c0d4a10 0x7f6c9c1563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:24.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.181+0000 7f6ca36c1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c9c15e4c0 0x7f6c9c0cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:24.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.181+0000 7f6ca36c1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c0b30a0 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.181+0000 7f6ca36c1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c0b2f20 con 0x7f6c9c0d4a10 2024-04-25T01:22:24.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.181+0000 7f6ca36c1640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6c9c15e8a0 con 0x7f6c9c15e4c0 2024-04-25T01:22:24.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.181+0000 7f6ca145d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c9c0b3ee0 0x7f6c9c0b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:24.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca145d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c9c0b3ee0 0x7f6c9c0b89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45486/0 (socket says 172.21.15.77:45486) 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca145d640 1 -- 172.21.15.77:0/1815923981 learned_addr learned my addr 172.21.15.77:0/1815923981 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca0c5c640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c9c0d4a10 0x7f6c9c1563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca1c5e640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c9c15e4c0 0x7f6c9c0cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca145d640 1 -- 172.21.15.77:0/1815923981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c9c15e4c0 msgr2=0x7f6c9c0cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca145d640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c9c15e4c0 0x7f6c9c0cd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca145d640 1 -- 172.21.15.77:0/1815923981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c9c0d4a10 msgr2=0x7f6c9c1563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca145d640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c9c0d4a10 0x7f6c9c1563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca145d640 1 -- 172.21.15.77:0/1815923981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9c0b7940 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca0c5c640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c9c0d4a10 0x7f6c9c1563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca145d640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c9c0b3ee0 0x7f6c9c0b89e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00c400 tx=0x7f6c8c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6c91ffb640 1 -- 172.21.15.77:0/1815923981 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0052d0 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6c91ffb640 1 -- 172.21.15.77:0/1815923981 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c8c005490 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6c91ffb640 1 -- 172.21.15.77:0/1815923981 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c007260 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca36c1640 1 -- 172.21.15.77:0/1815923981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9c0b7c20 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.182+0000 7f6ca36c1640 1 -- 172.21.15.77:0/1815923981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c9c0b84c0 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.184+0000 7f6ca36c1640 1 -- 172.21.15.77:0/1815923981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c9c068fb0 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.184+0000 7f6c91ffb640 1 -- 172.21.15.77:0/1815923981 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6c8c007400 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.184+0000 7f6c91ffb640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c8006be30 0x7f6c8006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:24.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.184+0000 7f6c91ffb640 1 -- 172.21.15.77:0/1815923981 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6c8c08a730 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.184+0000 7f6ca0c5c640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c8006be30 0x7f6c8006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:24.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.185+0000 7f6ca0c5c640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c8006be30 0x7f6c8006e2f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6c98006fd0 tx=0x7f6c98008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:24.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.189+0000 7f6c91ffb640 1 -- 172.21.15.77:0/1815923981 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6c8c058da0 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.399+0000 7f6ca36c1640 1 -- 172.21.15.77:0/1815923981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6c9c0b7df0 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.399+0000 7f6c91ffb640 1 -- 172.21.15.77:0/1815923981 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f6c8c0586a0 con 0x7f6c9c0b3ee0 2024-04-25T01:22:24.380 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:22:24.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 -- 172.21.15.77:0/1815923981 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c8006be30 msgr2=0x7f6c8006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:24.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c8006be30 0x7f6c8006e2f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6c98006fd0 tx=0x7f6c98008040 comp rx=0 tx=0).stop 2024-04-25T01:22:24.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 -- 172.21.15.77:0/1815923981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c9c0b3ee0 msgr2=0x7f6c9c0b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:24.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c9c0b3ee0 0x7f6c9c0b89e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00c400 tx=0x7f6c8c00c8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:24.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 -- 172.21.15.77:0/1815923981 shutdown_connections 2024-04-25T01:22:24.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c8006be30 0x7f6c8006e2f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:24.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c9c15e4c0 0x7f6c9c0cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:24.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c9c0d4a10 0x7f6c9c1563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:24.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 --2- 172.21.15.77:0/1815923981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c9c0b3ee0 0x7f6c9c0b89e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:24.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 -- 172.21.15.77:0/1815923981 >> 172.21.15.77:0/1815923981 conn(0x7f6c9c0d3ef0 msgr2=0x7f6c9c0d2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:24.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 -- 172.21.15.77:0/1815923981 shutdown_connections 2024-04-25T01:22:24.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:24.401+0000 7f6ca36c1640 1 -- 172.21.15.77:0/1815923981 wait complete. 2024-04-25T01:22:24.392 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:22:24.392 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:22:24.392 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi121.front.sepia.ceph.com 2024-04-25T01:22:24.392 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-25T01:22:24.392 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T01:22:24.392 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:22:24.393 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:22:24.393 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:22:24.393 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-25T01:22:24.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:24.393 DEBUG:teuthology.orchestra.run.smithi121:> ip addr 2024-04-25T01:22:24.408 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:22:24.408 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:22:24.408 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:82 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f0 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:83 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f1 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a2 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a3 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout: inet 172.21.15.121/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T01:22:24.409 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft 42706sec preferred_lft 42706sec 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::5365:1d8c:566a:b419/64 scope link noprefixroute 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether c6:d6:f8:5a:43:33 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::437:59ff:fe7c:eb05/64 scope link 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether c6:d6:f8:5a:43:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::c4d6:f8ff:fe5a:4333/64 scope link tentative 2024-04-25T01:22:24.410 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:24.411 DEBUG:teuthology.orchestra.run.smithi121:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:22:24.472 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-04-25T01:22:24.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:24.473 DEBUG:teuthology.orchestra.run.smithi121:> route 2024-04-25T01:22:24.529 INFO:teuthology.orchestra.run.smithi121.stdout:Kernel IP routing table 2024-04-25T01:22:24.529 INFO:teuthology.orchestra.run.smithi121.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:22:24.529 INFO:teuthology.orchestra.run.smithi121.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:22:24.529 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:22:24.529 INFO:teuthology.orchestra.run.smithi121.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:22:24.529 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:24.530 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-25T01:22:24.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:22:24.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:22:24.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:22:24.530 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-25T01:22:24.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:24.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:24.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:24.652 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-25T01:22:24.666 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T01:22:24.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:24.667 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list-id 2024-04-25T01:22:24.721 INFO:teuthology.orchestra.run.smithi121.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-25T01:22:24.722 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:24.722 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-25T01:22:24.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T01:22:24.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-25T01:22:24.722 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-25T01:22:24.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:24.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:24.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:24.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T01:22:24.860 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:22:24.860 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:22:24.860 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:22:24.860 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.860 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2024-04-25T01:22:24.860 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.860 INFO:teuthology.orchestra.run.smithi121.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:22:24.860 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 32:a1:22:6c:ee:bd brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T01:22:24.861 INFO:teuthology.orchestra.run.smithi121.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T01:22:24.861 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.861 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::30a1:22ff:fe6c:eebd/64 scope link tentative 2024-04-25T01:22:24.861 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:24.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T01:22:24.925 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:22:24.925 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:22:24.925 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:22:24.925 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.925 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2024-04-25T01:22:24.925 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.925 INFO:teuthology.orchestra.run.smithi121.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:22:24.925 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 32:a1:22:6c:ee:bd brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T01:22:24.926 INFO:teuthology.orchestra.run.smithi121.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T01:22:24.926 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.926 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::30a1:22ff:fe6c:eebd/64 scope link tentative 2024-04-25T01:22:24.926 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:22:24.927 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-25T01:22:24.927 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:24.927 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-25T01:22:24.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-25T01:22:24.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T01:22:24.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-25T01:22:24.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-25T01:22:24.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-25T01:22:24.928 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-25T01:22:25.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:25.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:25.074 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:22:25.074 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-25T01:22:25.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set brx.1 up 2024-04-25T01:22:25.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T01:22:25.074 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-25T01:22:25.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:25.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:25.149 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:22:25.149 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-25T01:22:25.163 INFO:teuthology.orchestra.run.smithi121.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-25T01:22:25.164 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:22:25.164 DEBUG:teuthology.orchestra.run.smithi121:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-25T01:22:25.218 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:22:25.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:25.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-25T01:22:25.301 INFO:teuthology.orchestra.run.smithi121.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-25T01:22:25.302 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-25T01:22:25.302 INFO:teuthology.orchestra.run.smithi121.stdout:invalid new device string format 2024-04-25T01:22:25.310 INFO:teuthology.orchestra.run.smithi121.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-25T01:22:25.310 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-25T01:22:25.310 INFO:teuthology.orchestra.run.smithi121.stdout:invalid new device string format 2024-04-25T01:22:25.310 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:22:25.310 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:22:25.315 INFO:teuthology.orchestra.run.smithi121.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-25T01:22:25.315 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-25T01:22:25.315 INFO:teuthology.orchestra.run.smithi121.stdout:invalid new device string format 2024-04-25T01:22:25.315 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:22:25.315 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:22:25.315 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790" 2024-04-25T01:22:25.315 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790:/ 2024-04-25T01:22:25.315 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=7c3c39ee-077e-476c-8994-eede4fab301d" will pass to kernel 2024-04-25T01:22:25.316 INFO:teuthology.orchestra.run.smithi121.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-04-25T01:22:25.316 INFO:teuthology.orchestra.run.smithi121.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:22:25.316 INFO:teuthology.orchestra.run.smithi121.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:22:25.316 INFO:teuthology.orchestra.run.smithi121.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:22:25.316 INFO:teuthology.orchestra.run.smithi121.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:22:25.317 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:22:25.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:25.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-25T01:22:25.381 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:25.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:25.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:25.477 INFO:teuthology.orchestra.run.smithi121.stdout:client4708 2024-04-25T01:22:25.478 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:22:25.478 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:25.493 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:25.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:25.587 INFO:teuthology.orchestra.run.smithi121.stdout:client4708 2024-04-25T01:22:25.588 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:22:25.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:22:25.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:25.642 INFO:teuthology.orchestra.run.smithi121.stdout:client4708 2024-04-25T01:22:25.643 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/7c3c39ee-077e-476c-8994-eede4fab301d.client4708/status') 2024-04-25T01:22:25.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:22:25.699 INFO:teuthology.orchestra.run.smithi121.stdout:instance: client.4708 (3)192.168.0.2:0/1576515534 2024-04-25T01:22:25.700 INFO:teuthology.orchestra.run.smithi121.stdout:blocklisted: false 2024-04-25T01:22:25.700 INFO:teuthology.orchestra.run.smithi121.stderr:0+1 records in 2024-04-25T01:22:25.700 INFO:teuthology.orchestra.run.smithi121.stderr:0+1 records out 2024-04-25T01:22:25.700 INFO:teuthology.orchestra.run.smithi121.stderr:69 bytes copied, 3.8664e-05 s, 1.8 MB/s 2024-04-25T01:22:25.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-25T01:22:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:22:25.708 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-25T01:22:25.729 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:25.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.858+0000 7f2af1563640 1 Processor -- start 2024-04-25T01:22:25.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.858+0000 7f2af1563640 1 -- start start 2024-04-25T01:22:25.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.858+0000 7f2af1563640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2aec162260 0x7f2aec0b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:25.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.858+0000 7f2af1563640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aec0d49d0 con 0x7f2aec161b10 2024-04-25T01:22:25.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.858+0000 7f2af1563640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aec0d4cb0 con 0x7f2aec162640 2024-04-25T01:22:25.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.858+0000 7f2af1563640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2aec0cee60 con 0x7f2aec162260 2024-04-25T01:22:25.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.859+0000 7f2aeb7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2aec161b10 0x7f2aec0cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36636/0 (socket says 172.21.15.77:36636) 2024-04-25T01:22:25.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.859+0000 7f2aeb7fe640 1 -- 172.21.15.77:0/4078958981 learned_addr learned my addr 172.21.15.77:0/4078958981 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:25.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.859+0000 7f2aeaffd640 1 --2- 172.21.15.77:0/4078958981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2aec162260 0x7f2aec0b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:25.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.859+0000 7f2ae9ffb640 1 -- 172.21.15.77:0/4078958981 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026407311 0 0) 0x7f2aec0d49d0 con 0x7f2aec161b10 2024-04-25T01:22:25.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.859+0000 7f2ae9ffb640 1 -- 172.21.15.77:0/4078958981 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ad4003520 con 0x7f2aec161b10 2024-04-25T01:22:25.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.859+0000 7f2ae9ffb640 1 -- 172.21.15.77:0/4078958981 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4092258055 0 0) 0x7f2aec0d4cb0 con 0x7f2aec162640 2024-04-25T01:22:25.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.859+0000 7f2ae9ffb640 1 -- 172.21.15.77:0/4078958981 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aec0d49d0 con 0x7f2aec162640 2024-04-25T01:22:25.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.859+0000 7f2aeaffd640 1 -- 172.21.15.77:0/4078958981 >> v1:172.21.15.121:6789/0 conn(0x7f2aec162640 legacy=0x7f2aec161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:25.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.859+0000 7f2aeaffd640 1 -- 172.21.15.77:0/4078958981 >> v1:172.21.15.77:6789/0 conn(0x7f2aec161b10 legacy=0x7f2aec0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:25.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.859+0000 7f2aeaffd640 1 -- 172.21.15.77:0/4078958981 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aec0cf140 con 0x7f2aec162260 2024-04-25T01:22:25.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.860+0000 7f2aeaffd640 1 --2- 172.21.15.77:0/4078958981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2aec162260 0x7f2aec0b0bd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0097d0 tx=0x7f2adc02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cda0e4d8c5b82cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:25.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.860+0000 7f2ae9ffb640 1 -- 172.21.15.77:0/4078958981 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc037070 con 0x7f2aec162260 2024-04-25T01:22:25.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.860+0000 7f2ae9ffb640 1 -- 172.21.15.77:0/4078958981 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2adc032440 con 0x7f2aec162260 2024-04-25T01:22:25.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.860+0000 7f2ae9ffb640 1 -- 172.21.15.77:0/4078958981 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc032810 con 0x7f2aec162260 2024-04-25T01:22:25.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.860+0000 7f2af1563640 1 -- 172.21.15.77:0/4078958981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2aec162260 msgr2=0x7f2aec0b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:25.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.860+0000 7f2af1563640 1 --2- 172.21.15.77:0/4078958981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2aec162260 0x7f2aec0b0bd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0097d0 tx=0x7f2adc02a210 comp rx=0 tx=0).stop 2024-04-25T01:22:25.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.861+0000 7f2af1563640 1 -- 172.21.15.77:0/4078958981 shutdown_connections 2024-04-25T01:22:25.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.861+0000 7f2af1563640 1 --2- 172.21.15.77:0/4078958981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2aec162260 0x7f2aec0b0bd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.861+0000 7f2af1563640 1 -- 172.21.15.77:0/4078958981 >> 172.21.15.77:0/4078958981 conn(0x7f2aec09d200 msgr2=0x7f2aec0d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:25.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.861+0000 7f2af1563640 1 -- 172.21.15.77:0/4078958981 shutdown_connections 2024-04-25T01:22:25.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.861+0000 7f2af1563640 1 -- 172.21.15.77:0/4078958981 wait complete. 2024-04-25T01:22:25.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.862+0000 7f2af1563640 1 Processor -- start 2024-04-25T01:22:25.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.862+0000 7f2af1563640 1 -- start start 2024-04-25T01:22:25.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2af1563640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aec161b10 0x7f2aec0c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:25.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2af1563640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec162260 0x7f2aec0ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:25.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2af1563640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2aec162640 0x7f2aec15b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:25.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2af1563640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec0cec20 con 0x7f2aec162260 2024-04-25T01:22:25.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2af1563640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec0ceaa0 con 0x7f2aec161b10 2024-04-25T01:22:25.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2af1563640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2aec0b1110 con 0x7f2aec162640 2024-04-25T01:22:25.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aea7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec162260 0x7f2aec0ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:25.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aea7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec162260 0x7f2aec0ca170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45492/0 (socket says 172.21.15.77:45492) 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aea7fc640 1 -- 172.21.15.77:0/3848319585 learned_addr learned my addr 172.21.15.77:0/3848319585 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aeaffd640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aec161b10 0x7f2aec0c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aeb7fe640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2aec162640 0x7f2aec15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aea7fc640 1 -- 172.21.15.77:0/3848319585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2aec162640 msgr2=0x7f2aec15b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aea7fc640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2aec162640 0x7f2aec15b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aea7fc640 1 -- 172.21.15.77:0/3848319585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aec161b10 msgr2=0x7f2aec0c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aea7fc640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aec161b10 0x7f2aec0c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aea7fc640 1 -- 172.21.15.77:0/3848319585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aec0c90d0 con 0x7f2aec162260 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aeaffd640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aec161b10 0x7f2aec0c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2aea7fc640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec162260 0x7f2aec0ca170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2ae000c430 tx=0x7f2ae000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2ad37fe640 1 -- 172.21.15.77:0/3848319585 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0017070 con 0x7f2aec162260 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2ad37fe640 1 -- 172.21.15.77:0/3848319585 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ae00052f0 con 0x7f2aec162260 2024-04-25T01:22:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2af1563640 1 -- 172.21.15.77:0/3848319585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aec0c93b0 con 0x7f2aec162260 2024-04-25T01:22:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.863+0000 7f2ad37fe640 1 -- 172.21.15.77:0/3848319585 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0003830 con 0x7f2aec162260 2024-04-25T01:22:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.864+0000 7f2af1563640 1 -- 172.21.15.77:0/3848319585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aec0c9970 con 0x7f2aec162260 2024-04-25T01:22:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.865+0000 7f2af1563640 1 -- 172.21.15.77:0/3848319585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aec068fb0 con 0x7f2aec162260 2024-04-25T01:22:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.865+0000 7f2ad37fe640 1 -- 172.21.15.77:0/3848319585 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2ae001a070 con 0x7f2aec162260 2024-04-25T01:22:25.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.866+0000 7f2ad37fe640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ad806be30 0x7f2ad806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:25.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.866+0000 7f2aeaffd640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ad806be30 0x7f2ad806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:25.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.866+0000 7f2ad37fe640 1 -- 172.21.15.77:0/3848319585 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6743+0+0 (secure 0 0 0) 0x7f2ae008a520 con 0x7f2aec162260 2024-04-25T01:22:25.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.866+0000 7f2aeaffd640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ad806be30 0x7f2ad806e2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2adc034000 tx=0x7f2adc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:25.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:25.870+0000 7f2ad37fe640 1 -- 172.21.15.77:0/3848319585 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2ae0058500 con 0x7f2aec162260 2024-04-25T01:22:26.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.043+0000 7f2af1563640 1 -- 172.21.15.77:0/3848319585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2aec09d6e0 con 0x7f2aec162260 2024-04-25T01:22:26.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.044+0000 7f2ad37fe640 1 -- 172.21.15.77:0/3848319585 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18922 (secure 0 0 0) 0x7f2ae005c800 con 0x7f2aec162260 2024-04-25T01:22:26.025 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:26.025 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:25.283152+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:26.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 -- 172.21.15.77:0/3848319585 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ad806be30 msgr2=0x7f2ad806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ad806be30 0x7f2ad806e2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2adc034000 tx=0x7f2adc02e000 comp rx=0 tx=0).stop 2024-04-25T01:22:26.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 -- 172.21.15.77:0/3848319585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec162260 msgr2=0x7f2aec0ca170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec162260 0x7f2aec0ca170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2ae000c430 tx=0x7f2ae000c8f0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 -- 172.21.15.77:0/3848319585 shutdown_connections 2024-04-25T01:22:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ad806be30 0x7f2ad806e2f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec162260 0x7f2aec0ca170 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2aec162640 0x7f2aec15b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 --2- 172.21.15.77:0/3848319585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aec161b10 0x7f2aec0c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 -- 172.21.15.77:0/3848319585 >> 172.21.15.77:0/3848319585 conn(0x7f2aec09d200 msgr2=0x7f2aec1611d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 -- 172.21.15.77:0/3848319585 shutdown_connections 2024-04-25T01:22:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.046+0000 7f2af1563640 1 -- 172.21.15.77:0/3848319585 wait complete. 2024-04-25T01:22:26.038 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:26.039 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:26.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.159+0000 7f01fac6b640 1 Processor -- start 2024-04-25T01:22:26.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.159+0000 7f01fac6b640 1 -- start start 2024-04-25T01:22:26.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.159+0000 7f01fac6b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01f40b37c0 0x7f01f415bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.159+0000 7f01fac6b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f415c210 con 0x7f01f40b47d0 2024-04-25T01:22:26.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.159+0000 7f01fac6b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f40b2ce0 con 0x7f01f4158a70 2024-04-25T01:22:26.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.159+0000 7f01fac6b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f01f40b2fc0 con 0x7f01f40b37c0 2024-04-25T01:22:26.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.160+0000 7f01f9208640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01f40b37c0 0x7f01f415bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.160+0000 7f01f3fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f01f40b47d0 0x7f01f415b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36650/0 (socket says 172.21.15.77:36650) 2024-04-25T01:22:26.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.160+0000 7f01f3fff640 1 -- 172.21.15.77:0/192894935 learned_addr learned my addr 172.21.15.77:0/192894935 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:26.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.160+0000 7f01f9208640 1 -- 172.21.15.77:0/192894935 >> v1:172.21.15.121:6789/0 conn(0x7f01f4158a70 legacy=0x7f01f40b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:26.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.160+0000 7f01f9208640 1 -- 172.21.15.77:0/192894935 >> v1:172.21.15.77:6789/0 conn(0x7f01f40b47d0 legacy=0x7f01f415b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.160+0000 7f01f9208640 1 -- 172.21.15.77:0/192894935 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f40b32a0 con 0x7f01f40b37c0 2024-04-25T01:22:26.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.160+0000 7f01f9208640 1 --2- 172.21.15.77:0/192894935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f40b37c0 0x7f01f415bc30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0097d0 tx=0x7f01dc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623a7bb3d2364963 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:26.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.161+0000 7f01f37fe640 1 -- 172.21.15.77:0/192894935 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc037070 con 0x7f01f40b37c0 2024-04-25T01:22:26.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.161+0000 7f01f37fe640 1 -- 172.21.15.77:0/192894935 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01dc032440 con 0x7f01f40b37c0 2024-04-25T01:22:26.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.161+0000 7f01f37fe640 1 -- 172.21.15.77:0/192894935 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc032810 con 0x7f01f40b37c0 2024-04-25T01:22:26.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.161+0000 7f01fac6b640 1 -- 172.21.15.77:0/192894935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f40b37c0 msgr2=0x7f01f415bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.161+0000 7f01fac6b640 1 --2- 172.21.15.77:0/192894935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f40b37c0 0x7f01f415bc30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0097d0 tx=0x7f01dc02a210 comp rx=0 tx=0).stop 2024-04-25T01:22:26.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.161+0000 7f01fac6b640 1 -- 172.21.15.77:0/192894935 shutdown_connections 2024-04-25T01:22:26.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.161+0000 7f01fac6b640 1 --2- 172.21.15.77:0/192894935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f40b37c0 0x7f01f415bc30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.161+0000 7f01fac6b640 1 -- 172.21.15.77:0/192894935 >> 172.21.15.77:0/192894935 conn(0x7f01f406b720 msgr2=0x7f01f40b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:26.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.161+0000 7f01fac6b640 1 -- 172.21.15.77:0/192894935 shutdown_connections 2024-04-25T01:22:26.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.161+0000 7f01fac6b640 1 -- 172.21.15.77:0/192894935 wait complete. 2024-04-25T01:22:26.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.162+0000 7f01fac6b640 1 Processor -- start 2024-04-25T01:22:26.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.162+0000 7f01fac6b640 1 -- start start 2024-04-25T01:22:26.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.163+0000 7f01fac6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f40b37c0 0x7f01f41442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.163+0000 7f01fac6b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f40b47d0 0x7f01f4135350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.163+0000 7f01fac6b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4158a70 0x7f01f40b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.163+0000 7f01fac6b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f01f4070cd0 con 0x7f01f40b37c0 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.163+0000 7f01fac6b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f01f4070b50 con 0x7f01f4158a70 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.163+0000 7f01fac6b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f01f4070e50 con 0x7f01f40b47d0 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.163+0000 7f01f8a07640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f40b37c0 0x7f01f41442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.163+0000 7f01f8a07640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f40b37c0 0x7f01f41442f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45510/0 (socket says 172.21.15.77:45510) 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.163+0000 7f01f8a07640 1 -- 172.21.15.77:0/1524750097 learned_addr learned my addr 172.21.15.77:0/1524750097 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.163+0000 7f01f9208640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4158a70 0x7f01f40b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.163+0000 7f01f3fff640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f40b47d0 0x7f01f4135350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.164+0000 7f01f8a07640 1 -- 172.21.15.77:0/1524750097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f40b47d0 msgr2=0x7f01f4135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.164+0000 7f01f8a07640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f40b47d0 0x7f01f4135350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.164+0000 7f01f8a07640 1 -- 172.21.15.77:0/1524750097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4158a70 msgr2=0x7f01f40b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.164+0000 7f01f8a07640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4158a70 0x7f01f40b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.164+0000 7f01f8a07640 1 -- 172.21.15.77:0/1524750097 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f4135890 con 0x7f01f40b37c0 2024-04-25T01:22:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.164+0000 7f01f8a07640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f40b37c0 0x7f01f41442f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f01ec00ba50 tx=0x7f01ec00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:26.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.164+0000 7f01f17fa640 1 -- 172.21.15.77:0/1524750097 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec019070 con 0x7f01f40b37c0 2024-04-25T01:22:26.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.164+0000 7f01f17fa640 1 -- 172.21.15.77:0/1524750097 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01ec00cc20 con 0x7f01f40b37c0 2024-04-25T01:22:26.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.164+0000 7f01f17fa640 1 -- 172.21.15.77:0/1524750097 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec014680 con 0x7f01f40b37c0 2024-04-25T01:22:26.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.164+0000 7f01fac6b640 1 -- 172.21.15.77:0/1524750097 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f4140390 con 0x7f01f40b37c0 2024-04-25T01:22:26.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.164+0000 7f01fac6b640 1 -- 172.21.15.77:0/1524750097 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f4140950 con 0x7f01f40b37c0 2024-04-25T01:22:26.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.165+0000 7f01fac6b640 1 -- 172.21.15.77:0/1524750097 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01f4068fb0 con 0x7f01f40b37c0 2024-04-25T01:22:26.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.165+0000 7f01f17fa640 1 -- 172.21.15.77:0/1524750097 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f01ec01c070 con 0x7f01f40b37c0 2024-04-25T01:22:26.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.166+0000 7f01f17fa640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01d406be30 0x7f01d406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.166+0000 7f01f3fff640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01d406be30 0x7f01d406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.166+0000 7f01f17fa640 1 -- 172.21.15.77:0/1524750097 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6743+0+0 (secure 0 0 0) 0x7f01ec08aed0 con 0x7f01f40b37c0 2024-04-25T01:22:26.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.166+0000 7f01f3fff640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01d406be30 0x7f01d406e2f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f01f40cc070 tx=0x7f01e4007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:26.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.170+0000 7f01f17fa640 1 -- 172.21.15.77:0/1524750097 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f01ec058eb0 con 0x7f01f40b37c0 2024-04-25T01:22:26.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.346+0000 7f01fac6b640 1 -- 172.21.15.77:0/1524750097 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f01f40b29c0 con 0x7f01f40b37c0 2024-04-25T01:22:26.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.346+0000 7f01f17fa640 1 -- 172.21.15.77:0/1524750097 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f01ec05d1b0 con 0x7f01f40b37c0 2024-04-25T01:22:26.327 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:26.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.348+0000 7f01fac6b640 1 -- 172.21.15.77:0/1524750097 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01d406be30 msgr2=0x7f01d406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.348+0000 7f01fac6b640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01d406be30 0x7f01d406e2f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f01f40cc070 tx=0x7f01e4007e80 comp rx=0 tx=0).stop 2024-04-25T01:22:26.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.348+0000 7f01fac6b640 1 -- 172.21.15.77:0/1524750097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f40b37c0 msgr2=0x7f01f41442f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.348+0000 7f01fac6b640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f40b37c0 0x7f01f41442f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f01ec00ba50 tx=0x7f01ec00bf10 comp rx=0 tx=0).stop 2024-04-25T01:22:26.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.348+0000 7f01fac6b640 1 -- 172.21.15.77:0/1524750097 shutdown_connections 2024-04-25T01:22:26.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.348+0000 7f01fac6b640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01d406be30 0x7f01d406e2f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.349+0000 7f01fac6b640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4158a70 0x7f01f40b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.349+0000 7f01fac6b640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f40b47d0 0x7f01f4135350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.349+0000 7f01fac6b640 1 --2- 172.21.15.77:0/1524750097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f40b37c0 0x7f01f41442f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.349+0000 7f01fac6b640 1 -- 172.21.15.77:0/1524750097 >> 172.21.15.77:0/1524750097 conn(0x7f01f406b720 msgr2=0x7f01f4071910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:26.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.349+0000 7f01fac6b640 1 -- 172.21.15.77:0/1524750097 shutdown_connections 2024-04-25T01:22:26.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.349+0000 7f01fac6b640 1 -- 172.21.15.77:0/1524750097 wait complete. 2024-04-25T01:22:26.340 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:26.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.459+0000 7f72b3af5640 1 Processor -- start 2024-04-25T01:22:26.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.459+0000 7f72b3af5640 1 -- start start 2024-04-25T01:22:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b3af5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72ac06d100 0x7f72ac16d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b3af5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72ac0b3c70 con 0x7f72ac162d10 2024-04-25T01:22:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b3af5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72ac0b3f50 con 0x7f72ac14c0f0 2024-04-25T01:22:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b3af5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f72ac0b3a30 con 0x7f72ac06d100 2024-04-25T01:22:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b2092640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72ac06d100 0x7f72ac16d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b1090640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f72ac162d10 0x7f72ac06c9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36664/0 (socket says 172.21.15.77:36664) 2024-04-25T01:22:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b1090640 1 -- 172.21.15.77:0/665816806 learned_addr learned my addr 172.21.15.77:0/665816806 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b088f640 1 -- 172.21.15.77:0/665816806 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208291908 0 0) 0x7f72ac0b3c70 con 0x7f72ac162d10 2024-04-25T01:22:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b088f640 1 -- 172.21.15.77:0/665816806 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7298003520 con 0x7f72ac162d10 2024-04-25T01:22:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b2092640 1 -- 172.21.15.77:0/665816806 >> v1:172.21.15.121:6789/0 conn(0x7f72ac14c0f0 legacy=0x7f72ac0b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b2092640 1 -- 172.21.15.77:0/665816806 >> v1:172.21.15.77:6789/0 conn(0x7f72ac162d10 legacy=0x7f72ac06c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.460+0000 7f72b2092640 1 -- 172.21.15.77:0/665816806 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ac0b4230 con 0x7f72ac06d100 2024-04-25T01:22:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.461+0000 7f72b2092640 1 --2- 172.21.15.77:0/665816806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ac06d100 0x7f72ac16d340 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009d90 tx=0x7f72a402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c8b9a5b7070d0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.461+0000 7f72b088f640 1 -- 172.21.15.77:0/665816806 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4031600 con 0x7f72ac06d100 2024-04-25T01:22:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.461+0000 7f72b088f640 1 -- 172.21.15.77:0/665816806 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72a40317a0 con 0x7f72ac06d100 2024-04-25T01:22:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.461+0000 7f72b088f640 1 -- 172.21.15.77:0/665816806 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4031b70 con 0x7f72ac06d100 2024-04-25T01:22:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.461+0000 7f72b3af5640 1 -- 172.21.15.77:0/665816806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ac06d100 msgr2=0x7f72ac16d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.461+0000 7f72b3af5640 1 --2- 172.21.15.77:0/665816806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ac06d100 0x7f72ac16d340 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009d90 tx=0x7f72a402a610 comp rx=0 tx=0).stop 2024-04-25T01:22:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.462+0000 7f72b3af5640 1 -- 172.21.15.77:0/665816806 shutdown_connections 2024-04-25T01:22:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.462+0000 7f72b3af5640 1 --2- 172.21.15.77:0/665816806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ac06d100 0x7f72ac16d340 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.462+0000 7f72b3af5640 1 -- 172.21.15.77:0/665816806 >> 172.21.15.77:0/665816806 conn(0x7f72ac06b720 msgr2=0x7f72ac0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.462+0000 7f72b3af5640 1 -- 172.21.15.77:0/665816806 shutdown_connections 2024-04-25T01:22:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.462+0000 7f72b3af5640 1 -- 172.21.15.77:0/665816806 wait complete. 2024-04-25T01:22:26.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.463+0000 7f72b3af5640 1 Processor -- start 2024-04-25T01:22:26.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.463+0000 7f72b3af5640 1 -- start start 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.463+0000 7f72b3af5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72ac06d100 0x7f72ac145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.463+0000 7f72b3af5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ac14c0f0 0x7f72ac15fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.463+0000 7f72b3af5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72ac162d10 0x7f72ac15ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.463+0000 7f72b3af5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac071c40 con 0x7f72ac14c0f0 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.463+0000 7f72b3af5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac0b47f0 con 0x7f72ac162d10 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.463+0000 7f72b3af5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72ac071dc0 con 0x7f72ac06d100 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.463+0000 7f72b1090640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ac14c0f0 0x7f72ac15fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b1090640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ac14c0f0 0x7f72ac15fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45532/0 (socket says 172.21.15.77:45532) 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b1090640 1 -- 172.21.15.77:0/950199933 learned_addr learned my addr 172.21.15.77:0/950199933 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b1891640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72ac06d100 0x7f72ac145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b2092640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72ac162d10 0x7f72ac15ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b1090640 1 -- 172.21.15.77:0/950199933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72ac06d100 msgr2=0x7f72ac145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b1090640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72ac06d100 0x7f72ac145bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b1090640 1 -- 172.21.15.77:0/950199933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72ac162d10 msgr2=0x7f72ac15ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b1090640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72ac162d10 0x7f72ac15ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b1090640 1 -- 172.21.15.77:0/950199933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ac160730 con 0x7f72ac14c0f0 2024-04-25T01:22:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b1891640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72ac06d100 0x7f72ac145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b1090640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ac14c0f0 0x7f72ac15fc70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c3c0 tx=0x7f729c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72a27fc640 1 -- 172.21.15.77:0/950199933 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c017070 con 0x7f72ac14c0f0 2024-04-25T01:22:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72a27fc640 1 -- 172.21.15.77:0/950199933 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f729c005ce0 con 0x7f72ac14c0f0 2024-04-25T01:22:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b3af5640 1 -- 172.21.15.77:0/950199933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72ac160a10 con 0x7f72ac14c0f0 2024-04-25T01:22:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72a27fc640 1 -- 172.21.15.77:0/950199933 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c0072f0 con 0x7f72ac14c0f0 2024-04-25T01:22:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.464+0000 7f72b3af5640 1 -- 172.21.15.77:0/950199933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72ac160fd0 con 0x7f72ac14c0f0 2024-04-25T01:22:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.465+0000 7f72b3af5640 1 -- 172.21.15.77:0/950199933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72ac068fb0 con 0x7f72ac14c0f0 2024-04-25T01:22:26.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.466+0000 7f72a27fc640 1 -- 172.21.15.77:0/950199933 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f729c01a070 con 0x7f72ac14c0f0 2024-04-25T01:22:26.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.466+0000 7f72a27fc640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729006be30 0x7f729006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.466+0000 7f72a27fc640 1 -- 172.21.15.77:0/950199933 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f729c08a670 con 0x7f72ac14c0f0 2024-04-25T01:22:26.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.466+0000 7f72b1891640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729006be30 0x7f729006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.467+0000 7f72b1891640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729006be30 0x7f729006e2f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f72a8003eb0 tx=0x7f72a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:26.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.470+0000 7f72a27fc640 1 -- 172.21.15.77:0/950199933 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f729c058c10 con 0x7f72ac14c0f0 2024-04-25T01:22:26.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.647+0000 7f72b3af5640 1 -- 172.21.15.77:0/950199933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f72ac146630 con 0x7f72ac14c0f0 2024-04-25T01:22:26.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.647+0000 7f72a27fc640 1 -- 172.21.15.77:0/950199933 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f729c058510 con 0x7f72ac14c0f0 2024-04-25T01:22:26.629 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:26.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 -- 172.21.15.77:0/950199933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729006be30 msgr2=0x7f729006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729006be30 0x7f729006e2f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f72a8003eb0 tx=0x7f72a8009040 comp rx=0 tx=0).stop 2024-04-25T01:22:26.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 -- 172.21.15.77:0/950199933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ac14c0f0 msgr2=0x7f72ac15fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ac14c0f0 0x7f72ac15fc70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c3c0 tx=0x7f729c00c880 comp rx=0 tx=0).stop 2024-04-25T01:22:26.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 -- 172.21.15.77:0/950199933 shutdown_connections 2024-04-25T01:22:26.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f729006be30 0x7f729006e2f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ac14c0f0 0x7f72ac15fc70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72ac162d10 0x7f72ac15ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 --2- 172.21.15.77:0/950199933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72ac06d100 0x7f72ac145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 -- 172.21.15.77:0/950199933 >> 172.21.15.77:0/950199933 conn(0x7f72ac06b720 msgr2=0x7f72ac0b3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:26.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 -- 172.21.15.77:0/950199933 shutdown_connections 2024-04-25T01:22:26.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.649+0000 7f72b3af5640 1 -- 172.21.15.77:0/950199933 wait complete. 2024-04-25T01:22:26.642 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:26.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.760+0000 7f327548c640 1 Processor -- start 2024-04-25T01:22:26.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.760+0000 7f327548c640 1 -- start start 2024-04-25T01:22:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f327548c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f32700d23d0 0x7f32700d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f327548c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f32700b9a90 0x7f32700d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f327548c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32700cea40 con 0x7f32700d27b0 2024-04-25T01:22:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f327548c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f32700cece0 con 0x7f32700d23d0 2024-04-25T01:22:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f327548c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f32700cefc0 con 0x7f32700b9a90 2024-04-25T01:22:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f326effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f32700d23d0 0x7f32700d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f3267fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f32700d27b0 0x7f32700b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57364/0 (socket says 172.21.15.77:57364) 2024-04-25T01:22:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f3267fff640 1 -- 172.21.15.77:0/877749473 learned_addr learned my addr 172.21.15.77:0/877749473 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f326effd640 1 -- 172.21.15.77:0/877749473 >> v2:172.21.15.121:3300/0 conn(0x7f32700b9a90 msgr2=0x7f32700d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f326effd640 1 --2- 172.21.15.77:0/877749473 >> v2:172.21.15.121:3300/0 conn(0x7f32700b9a90 0x7f32700d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f326effd640 1 -- 172.21.15.77:0/877749473 >> v1:172.21.15.77:6789/0 conn(0x7f32700d27b0 legacy=0x7f32700b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f326effd640 1 -- 172.21.15.77:0/877749473 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32700cf2a0 con 0x7f32700d23d0 2024-04-25T01:22:26.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.761+0000 7f326e7fc640 1 -- 172.21.15.77:0/877749473 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1970198804 0 0) 0x7f32700cea40 con 0x7f32700d27b0 2024-04-25T01:22:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.762+0000 7f326effd640 1 --2- 172.21.15.77:0/877749473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32700d23d0 0x7f32700d4920 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f327006b140 tx=0x7f326002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5f30183f23f281 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.762+0000 7f326e7fc640 1 -- 172.21.15.77:0/877749473 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260037070 con 0x7f32700d23d0 2024-04-25T01:22:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.762+0000 7f326e7fc640 1 -- 172.21.15.77:0/877749473 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3260032440 con 0x7f32700d23d0 2024-04-25T01:22:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.762+0000 7f326e7fc640 1 -- 172.21.15.77:0/877749473 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260032810 con 0x7f32700d23d0 2024-04-25T01:22:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.763+0000 7f327548c640 1 -- 172.21.15.77:0/877749473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32700d23d0 msgr2=0x7f32700d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.763+0000 7f327548c640 1 --2- 172.21.15.77:0/877749473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32700d23d0 0x7f32700d4920 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f327006b140 tx=0x7f326002a230 comp rx=0 tx=0).stop 2024-04-25T01:22:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.763+0000 7f327548c640 1 -- 172.21.15.77:0/877749473 shutdown_connections 2024-04-25T01:22:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.763+0000 7f327548c640 1 --2- 172.21.15.77:0/877749473 >> v2:172.21.15.121:3300/0 conn(0x7f32700b9a90 0x7f32700d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.763+0000 7f327548c640 1 --2- 172.21.15.77:0/877749473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32700d23d0 0x7f32700d4920 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.763+0000 7f327548c640 1 -- 172.21.15.77:0/877749473 >> 172.21.15.77:0/877749473 conn(0x7f32700d3e30 msgr2=0x7f32700d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.763+0000 7f327548c640 1 -- 172.21.15.77:0/877749473 shutdown_connections 2024-04-25T01:22:26.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.763+0000 7f327548c640 1 -- 172.21.15.77:0/877749473 wait complete. 2024-04-25T01:22:26.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.764+0000 7f327548c640 1 Processor -- start 2024-04-25T01:22:26.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.764+0000 7f327548c640 1 -- start start 2024-04-25T01:22:26.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.764+0000 7f327548c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32700b9a90 0x7f327014b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.764+0000 7f327548c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32700d23d0 0x7f32700cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.764+0000 7f327548c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32700d27b0 0x7f32700cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.764+0000 7f327548c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f32700cfd80 con 0x7f32700d27b0 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f327548c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f32700cf860 con 0x7f32700b9a90 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f327548c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f32700b6c80 con 0x7f32700d23d0 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f326f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32700d27b0 0x7f32700cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f326f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32700d27b0 0x7f32700cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48830/0 (socket says 172.21.15.77:48830) 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f326f7fe640 1 -- 172.21.15.77:0/3285255777 learned_addr learned my addr 172.21.15.77:0/3285255777 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f326f7fe640 1 -- 172.21.15.77:0/3285255777 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32700d23d0 msgr2=0x7f32700cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f326f7fe640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32700d23d0 0x7f32700cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f326f7fe640 1 -- 172.21.15.77:0/3285255777 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32700b9a90 msgr2=0x7f327014b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f326f7fe640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32700b9a90 0x7f327014b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f326f7fe640 1 -- 172.21.15.77:0/3285255777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32701f5050 con 0x7f32700d27b0 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f326f7fe640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32700d27b0 0x7f32700cc490 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f326800b9a0 tx=0x7f326800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f32677fe640 1 -- 172.21.15.77:0/3285255777 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3268019070 con 0x7f32700d27b0 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f32677fe640 1 -- 172.21.15.77:0/3285255777 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f326800cbc0 con 0x7f32700d27b0 2024-04-25T01:22:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f32677fe640 1 -- 172.21.15.77:0/3285255777 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3268014440 con 0x7f32700d27b0 2024-04-25T01:22:26.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f327548c640 1 -- 172.21.15.77:0/3285255777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32701f7a60 con 0x7f32700d27b0 2024-04-25T01:22:26.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.765+0000 7f327548c640 1 -- 172.21.15.77:0/3285255777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f327014b6d0 con 0x7f32700d27b0 2024-04-25T01:22:26.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.766+0000 7f327548c640 1 -- 172.21.15.77:0/3285255777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3270068fb0 con 0x7f32700d27b0 2024-04-25T01:22:26.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.767+0000 7f32677fe640 1 -- 172.21.15.77:0/3285255777 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f326801c070 con 0x7f32700d27b0 2024-04-25T01:22:26.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.767+0000 7f32677fe640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f325006be30 0x7f325006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:26.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.767+0000 7f32677fe640 1 -- 172.21.15.77:0/3285255777 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f326808afc0 con 0x7f32700d27b0 2024-04-25T01:22:26.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.767+0000 7f326effd640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f325006be30 0x7f325006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:26.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.768+0000 7f326effd640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f325006be30 0x7f325006e2f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3260034000 tx=0x7f326002c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:26.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.771+0000 7f32677fe640 1 -- 172.21.15.77:0/3285255777 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3268059560 con 0x7f32700d27b0 2024-04-25T01:22:26.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.942+0000 7f327548c640 1 -- 172.21.15.77:0/3285255777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f32700d1dc0 con 0x7f32700d27b0 2024-04-25T01:22:26.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.942+0000 7f32677fe640 1 -- 172.21.15.77:0/3285255777 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f3268058e60 con 0x7f32700d27b0 2024-04-25T01:22:26.924 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 -- 172.21.15.77:0/3285255777 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f325006be30 msgr2=0x7f325006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f325006be30 0x7f325006e2f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3260034000 tx=0x7f326002c000 comp rx=0 tx=0).stop 2024-04-25T01:22:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 -- 172.21.15.77:0/3285255777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32700d27b0 msgr2=0x7f32700cc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32700d27b0 0x7f32700cc490 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f326800b9a0 tx=0x7f326800be60 comp rx=0 tx=0).stop 2024-04-25T01:22:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 -- 172.21.15.77:0/3285255777 shutdown_connections 2024-04-25T01:22:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f325006be30 0x7f325006e2f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32700d27b0 0x7f32700cc490 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32700d23d0 0x7f32700cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 --2- 172.21.15.77:0/3285255777 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32700b9a90 0x7f327014b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 -- 172.21.15.77:0/3285255777 >> 172.21.15.77:0/3285255777 conn(0x7f32700d3e30 msgr2=0x7f32700d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 -- 172.21.15.77:0/3285255777 shutdown_connections 2024-04-25T01:22:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:26.944+0000 7f327548c640 1 -- 172.21.15.77:0/3285255777 wait complete. 2024-04-25T01:22:26.937 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:27.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.053+0000 7f6f854d4640 1 Processor -- start 2024-04-25T01:22:27.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.054+0000 7f6f854d4640 1 -- start start 2024-04-25T01:22:27.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.054+0000 7f6f854d4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f800722e0 0x7f6f8006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.054+0000 7f6f854d4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f8015fc40 0x7f6f801600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.054+0000 7f6f854d4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f8006e410 con 0x7f6f800cedc0 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.054+0000 7f6f854d4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6f80160aa0 con 0x7f6f8015fc40 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.054+0000 7f6f854d4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6f8006da40 con 0x7f6f800722e0 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.054+0000 7f6f7e7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f8015fc40 0x7f6f801600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.055+0000 7f6f7f7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6f800cedc0 0x7f6f800cf250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57380/0 (socket says 172.21.15.77:57380) 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.055+0000 7f6f7f7fe640 1 -- 172.21.15.77:0/2003666421 learned_addr learned my addr 172.21.15.77:0/2003666421 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.055+0000 7f6f7effd640 1 --2- 172.21.15.77:0/2003666421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f800722e0 0x7f6f8006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.055+0000 7f6f7e7fc640 1 -- 172.21.15.77:0/2003666421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f800722e0 msgr2=0x7f6f8006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.055+0000 7f6f7e7fc640 1 --2- 172.21.15.77:0/2003666421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f800722e0 0x7f6f8006ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.055+0000 7f6f7e7fc640 1 -- 172.21.15.77:0/2003666421 >> v1:172.21.15.77:6789/0 conn(0x7f6f800cedc0 legacy=0x7f6f800cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.055+0000 7f6f7dffb640 1 -- 172.21.15.77:0/2003666421 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2297353178 0 0) 0x7f6f8006e410 con 0x7f6f800cedc0 2024-04-25T01:22:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.055+0000 7f6f7e7fc640 1 -- 172.21.15.77:0/2003666421 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8006d110 con 0x7f6f8015fc40 2024-04-25T01:22:27.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f7e7fc640 1 --2- 172.21.15.77:0/2003666421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8015fc40 0x7f6f801600c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6f740097a0 tx=0x7f6f7402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54d7fac10bfb979 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f7dffb640 1 -- 172.21.15.77:0/2003666421 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f74037070 con 0x7f6f8015fc40 2024-04-25T01:22:27.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f7dffb640 1 -- 172.21.15.77:0/2003666421 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f7402ae50 con 0x7f6f8015fc40 2024-04-25T01:22:27.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f7dffb640 1 -- 172.21.15.77:0/2003666421 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f74032660 con 0x7f6f8015fc40 2024-04-25T01:22:27.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f854d4640 1 -- 172.21.15.77:0/2003666421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8015fc40 msgr2=0x7f6f801600c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f854d4640 1 --2- 172.21.15.77:0/2003666421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8015fc40 0x7f6f801600c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6f740097a0 tx=0x7f6f7402a100 comp rx=0 tx=0).stop 2024-04-25T01:22:27.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f854d4640 1 -- 172.21.15.77:0/2003666421 shutdown_connections 2024-04-25T01:22:27.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f854d4640 1 --2- 172.21.15.77:0/2003666421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8015fc40 0x7f6f801600c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f854d4640 1 --2- 172.21.15.77:0/2003666421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f800722e0 0x7f6f8006ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f854d4640 1 -- 172.21.15.77:0/2003666421 >> 172.21.15.77:0/2003666421 conn(0x7f6f8006b720 msgr2=0x7f6f800b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f854d4640 1 -- 172.21.15.77:0/2003666421 shutdown_connections 2024-04-25T01:22:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.056+0000 7f6f854d4640 1 -- 172.21.15.77:0/2003666421 wait complete. 2024-04-25T01:22:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f854d4640 1 Processor -- start 2024-04-25T01:22:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f854d4640 1 -- start start 2024-04-25T01:22:27.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f854d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f800722e0 0x7f6f80157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f854d4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f800cedc0 0x7f6f800aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f854d4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8015fc40 0x7f6f800b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f854d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8006d6d0 con 0x7f6f800722e0 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f854d4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8006cec0 con 0x7f6f800cedc0 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f854d4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6f80070b50 con 0x7f6f8015fc40 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f7effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f800722e0 0x7f6f80157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f7effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f800722e0 0x7f6f80157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48842/0 (socket says 172.21.15.77:48842) 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f7effd640 1 -- 172.21.15.77:0/2313789416 learned_addr learned my addr 172.21.15.77:0/2313789416 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f7f7fe640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8015fc40 0x7f6f800b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.058+0000 7f6f7e7fc640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f800cedc0 0x7f6f800aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.059+0000 7f6f7effd640 1 -- 172.21.15.77:0/2313789416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8015fc40 msgr2=0x7f6f800b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.059+0000 7f6f7effd640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8015fc40 0x7f6f800b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.059+0000 7f6f7effd640 1 -- 172.21.15.77:0/2313789416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f800cedc0 msgr2=0x7f6f800aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.059+0000 7f6f7effd640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f800cedc0 0x7f6f800aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.059+0000 7f6f7effd640 1 -- 172.21.15.77:0/2313789416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f80157e00 con 0x7f6f800722e0 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.059+0000 7f6f7effd640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f800722e0 0x7f6f80157720 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6f7000d800 tx=0x7f6f7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.059+0000 7f6f677fe640 1 -- 172.21.15.77:0/2313789416 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7000b6c0 con 0x7f6f800722e0 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.059+0000 7f6f677fe640 1 -- 172.21.15.77:0/2313789416 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f70019070 con 0x7f6f800722e0 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.059+0000 7f6f854d4640 1 -- 172.21.15.77:0/2313789416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f801580e0 con 0x7f6f800722e0 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.059+0000 7f6f677fe640 1 -- 172.21.15.77:0/2313789416 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7000ba80 con 0x7f6f800722e0 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.059+0000 7f6f854d4640 1 -- 172.21.15.77:0/2313789416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f801586a0 con 0x7f6f800722e0 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.060+0000 7f6f854d4640 1 -- 172.21.15.77:0/2313789416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f80068fb0 con 0x7f6f800722e0 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.061+0000 7f6f677fe640 1 -- 172.21.15.77:0/2313789416 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6f7000bc20 con 0x7f6f800722e0 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.061+0000 7f6f677fe640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f6c06be30 0x7f6f6c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.061+0000 7f6f677fe640 1 -- 172.21.15.77:0/2313789416 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f6f7008b5f0 con 0x7f6f800722e0 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.061+0000 7f6f7e7fc640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f6c06be30 0x7f6f6c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.061+0000 7f6f7e7fc640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f6c06be30 0x7f6f6c06e2f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6f74034000 tx=0x7f6f7402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.065+0000 7f6f677fe640 1 -- 172.21.15.77:0/2313789416 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6f70059540 con 0x7f6f800722e0 2024-04-25T01:22:27.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.238+0000 7f6f854d4640 1 -- 172.21.15.77:0/2313789416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6f80072c90 con 0x7f6f800722e0 2024-04-25T01:22:27.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.239+0000 7f6f677fe640 1 -- 172.21.15.77:0/2313789416 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f6f7005d840 con 0x7f6f800722e0 2024-04-25T01:22:27.221 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 -- 172.21.15.77:0/2313789416 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f6c06be30 msgr2=0x7f6f6c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f6c06be30 0x7f6f6c06e2f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6f74034000 tx=0x7f6f7402e000 comp rx=0 tx=0).stop 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 -- 172.21.15.77:0/2313789416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f800722e0 msgr2=0x7f6f80157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f800722e0 0x7f6f80157720 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6f7000d800 tx=0x7f6f7000dcc0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 -- 172.21.15.77:0/2313789416 shutdown_connections 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f6c06be30 0x7f6f6c06e2f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8015fc40 0x7f6f800b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f800cedc0 0x7f6f800aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 --2- 172.21.15.77:0/2313789416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f800722e0 0x7f6f80157720 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 -- 172.21.15.77:0/2313789416 >> 172.21.15.77:0/2313789416 conn(0x7f6f8006b720 msgr2=0x7f6f8006e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 -- 172.21.15.77:0/2313789416 shutdown_connections 2024-04-25T01:22:27.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.241+0000 7f6f854d4640 1 -- 172.21.15.77:0/2313789416 wait complete. 2024-04-25T01:22:27.233 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:27.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.350+0000 7f8497a4d640 1 Processor -- start 2024-04-25T01:22:27.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.350+0000 7f8497a4d640 1 -- start start 2024-04-25T01:22:27.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8497a4d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f849006ed60 0x7f849015f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8497a4d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8490070e60 con 0x7f8490143d20 2024-04-25T01:22:27.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8497a4d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8490071140 con 0x7f8490143940 2024-04-25T01:22:27.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8497a4d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8490071420 con 0x7f849006ed60 2024-04-25T01:22:27.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8494fe8640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8490143d20 0x7f849015ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57396/0 (socket says 172.21.15.77:57396) 2024-04-25T01:22:27.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8494fe8640 1 -- 172.21.15.77:0/2732955539 learned_addr learned my addr 172.21.15.77:0/2732955539 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:27.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8495fea640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f849006ed60 0x7f849015f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1365021996 0 0) 0x7f8490070e60 con 0x7f8490143d20 2024-04-25T01:22:27.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f847c003520 con 0x7f8490143d20 2024-04-25T01:22:27.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2730406307 0 0) 0x7f847c003520 con 0x7f8490143d20 2024-04-25T01:22:27.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f849006ed60 msgr2=0x7f849015f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:27.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f849006ed60 0x7f849015f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 >> v1:172.21.15.121:6789/0 conn(0x7f8490143940 legacy=0x7f849006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8490071700 con 0x7f8490143d20 2024-04-25T01:22:27.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f8480001670 con 0x7f8490143d20 2024-04-25T01:22:27.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 >> v1:172.21.15.77:6789/0 conn(0x7f8490143d20 legacy=0x7f849015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c0039e0 0x7f847c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f849006ed60 0x7f847c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f847c004d40 0x7f847c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8490071420 con 0x7f847c004d40 2024-04-25T01:22:27.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f847c00dc80 con 0x7f847c0039e0 2024-04-25T01:22:27.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f847c00de80 con 0x7f849006ed60 2024-04-25T01:22:27.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.351+0000 7f8495fea640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f847c004d40 0x7f847c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.352+0000 7f8494fe8640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f849006ed60 0x7f847c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.352+0000 7f84957e9640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c0039e0 0x7f847c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.352+0000 7f8495fea640 1 -- 172.21.15.77:0/2732955539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f849006ed60 msgr2=0x7f847c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:27.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.352+0000 7f8495fea640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f849006ed60 0x7f847c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.352+0000 7f8495fea640 1 -- 172.21.15.77:0/2732955539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c0039e0 msgr2=0x7f847c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:27.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.352+0000 7f8495fea640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c0039e0 0x7f847c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.352+0000 7f8495fea640 1 -- 172.21.15.77:0/2732955539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847c003520 con 0x7f847c004d40 2024-04-25T01:22:27.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.352+0000 7f8495fea640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f847c004d40 0x7f847c0051c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f849006f160 tx=0x7f847c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39006429ad89c19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.353+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488037070 con 0x7f847c004d40 2024-04-25T01:22:27.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.353+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f848802e050 con 0x7f847c004d40 2024-04-25T01:22:27.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.353+0000 7f8487fff640 1 -- 172.21.15.77:0/2732955539 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488009c10 con 0x7f847c004d40 2024-04-25T01:22:27.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.353+0000 7f8497a4d640 1 -- 172.21.15.77:0/2732955539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f847c004d40 msgr2=0x7f847c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.353+0000 7f8497a4d640 1 --2- 172.21.15.77:0/2732955539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f847c004d40 0x7f847c0051c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f849006f160 tx=0x7f847c005830 comp rx=0 tx=0).stop 2024-04-25T01:22:27.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.353+0000 7f8495fea640 1 -- 172.21.15.77:0/2732955539 reap_dead start 2024-04-25T01:22:27.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.353+0000 7f8497a4d640 1 -- 172.21.15.77:0/2732955539 shutdown_connections 2024-04-25T01:22:27.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.353+0000 7f8497a4d640 1 -- 172.21.15.77:0/2732955539 >> 172.21.15.77:0/2732955539 conn(0x7f84900b1820 msgr2=0x7f84900b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:27.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.353+0000 7f8497a4d640 1 -- 172.21.15.77:0/2732955539 shutdown_connections 2024-04-25T01:22:27.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.353+0000 7f8497a4d640 1 -- 172.21.15.77:0/2732955539 wait complete. 2024-04-25T01:22:27.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.354+0000 7f8497a4d640 1 Processor -- start 2024-04-25T01:22:27.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.354+0000 7f8497a4d640 1 -- start start 2024-04-25T01:22:27.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8497a4d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c004d40 0x7f849013e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8497a4d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8490158bc0 0x7f84900ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8497a4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84901580e0 0x7f84900ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8497a4d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f849015f400 con 0x7f84901580e0 2024-04-25T01:22:27.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8497a4d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8490072e40 con 0x7f847c004d40 2024-04-25T01:22:27.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8497a4d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f849015f580 con 0x7f8490158bc0 2024-04-25T01:22:27.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8495fea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84901580e0 0x7f84900ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8495fea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84901580e0 0x7f84900ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48870/0 (socket says 172.21.15.77:48870) 2024-04-25T01:22:27.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8495fea640 1 -- 172.21.15.77:0/3271759706 learned_addr learned my addr 172.21.15.77:0/3271759706 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:27.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8494fe8640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8490158bc0 0x7f84900ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f84957e9640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c004d40 0x7f849013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8495fea640 1 -- 172.21.15.77:0/3271759706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8490158bc0 msgr2=0x7f84900ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8495fea640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8490158bc0 0x7f84900ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8495fea640 1 -- 172.21.15.77:0/3271759706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c004d40 msgr2=0x7f849013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8495fea640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c004d40 0x7f849013e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8495fea640 1 -- 172.21.15.77:0/3271759706 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84901534d0 con 0x7f84901580e0 2024-04-25T01:22:27.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.355+0000 7f8494fe8640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8490158bc0 0x7f84900ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:27.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.356+0000 7f8495fea640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84901580e0 0x7f84900ce7c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f848802ae80 tx=0x7f8488009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.356+0000 7f8485ffb640 1 -- 172.21.15.77:0/3271759706 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488043070 con 0x7f84901580e0 2024-04-25T01:22:27.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.356+0000 7f8485ffb640 1 -- 172.21.15.77:0/3271759706 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8488003990 con 0x7f84901580e0 2024-04-25T01:22:27.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.356+0000 7f8497a4d640 1 -- 172.21.15.77:0/3271759706 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8490153770 con 0x7f84901580e0 2024-04-25T01:22:27.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.356+0000 7f8485ffb640 1 -- 172.21.15.77:0/3271759706 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488037040 con 0x7f84901580e0 2024-04-25T01:22:27.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.356+0000 7f8497a4d640 1 -- 172.21.15.77:0/3271759706 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84900cc060 con 0x7f84901580e0 2024-04-25T01:22:27.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.357+0000 7f8497a4d640 1 -- 172.21.15.77:0/3271759706 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8490068fb0 con 0x7f84901580e0 2024-04-25T01:22:27.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.357+0000 7f8485ffb640 1 -- 172.21.15.77:0/3271759706 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f848803a070 con 0x7f84901580e0 2024-04-25T01:22:27.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.358+0000 7f8485ffb640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f847406be60 0x7f847406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.358+0000 7f84957e9640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f847406be60 0x7f847406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.358+0000 7f8485ffb640 1 -- 172.21.15.77:0/3271759706 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f84880aac40 con 0x7f84901580e0 2024-04-25T01:22:27.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.358+0000 7f84957e9640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f847406be60 0x7f847406e320 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f848c005590 tx=0x7f848c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.362+0000 7f8485ffb640 1 -- 172.21.15.77:0/3271759706 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8488078b90 con 0x7f84901580e0 2024-04-25T01:22:27.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.534+0000 7f8497a4d640 1 -- 172.21.15.77:0/3271759706 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f849006e2d0 con 0x7f84901580e0 2024-04-25T01:22:27.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.536+0000 7f8485ffb640 1 -- 172.21.15.77:0/3271759706 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18922 (secure 0 0 0) 0x7f848807ce90 con 0x7f84901580e0 2024-04-25T01:22:27.518 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:27.518 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:26.287128+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 -- 172.21.15.77:0/3271759706 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f847406be60 msgr2=0x7f847406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f847406be60 0x7f847406e320 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f848c005590 tx=0x7f848c009040 comp rx=0 tx=0).stop 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 -- 172.21.15.77:0/3271759706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84901580e0 msgr2=0x7f84900ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84901580e0 0x7f84900ce7c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f848802ae80 tx=0x7f8488009750 comp rx=0 tx=0).stop 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 -- 172.21.15.77:0/3271759706 shutdown_connections 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f847406be60 0x7f847406e320 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84901580e0 0x7f84900ce7c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8490158bc0 0x7f84900ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 --2- 172.21.15.77:0/3271759706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c004d40 0x7f849013e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 -- 172.21.15.77:0/3271759706 >> 172.21.15.77:0/3271759706 conn(0x7f84900b1820 msgr2=0x7f849015d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 -- 172.21.15.77:0/3271759706 shutdown_connections 2024-04-25T01:22:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.538+0000 7f8497a4d640 1 -- 172.21.15.77:0/3271759706 wait complete. 2024-04-25T01:22:27.530 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:27.531 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:27.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.649+0000 7f9d213e9640 1 Processor -- start 2024-04-25T01:22:27.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.649+0000 7f9d213e9640 1 -- start start 2024-04-25T01:22:27.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.650+0000 7f9d213e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d1c0b9a90 0x7f9d1c0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.650+0000 7f9d213e9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d1c0d27b0 0x7f9d1c0d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.650+0000 7f9d213e9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d1c0cea40 con 0x7f9d1c0d23d0 2024-04-25T01:22:27.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.650+0000 7f9d213e9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f9d1c0cece0 con 0x7f9d1c0b9a90 2024-04-25T01:22:27.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.650+0000 7f9d213e9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d1c0cefc0 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.650+0000 7f9d13fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d1c0b9a90 0x7f9d1c0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.650+0000 7f9d13fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d1c0b9a90 0x7f9d1c0d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48874/0 (socket says 172.21.15.77:48874) 2024-04-25T01:22:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.650+0000 7f9d13fff640 1 -- 172.21.15.77:0/2700445132 learned_addr learned my addr 172.21.15.77:0/2700445132 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.650+0000 7f9d1b7fe640 1 --2- 172.21.15.77:0/2700445132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d1c0d27b0 0x7f9d1c0d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.651+0000 7f9d13fff640 1 -- 172.21.15.77:0/2700445132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d1c0d27b0 msgr2=0x7f9d1c0d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.651+0000 7f9d13fff640 1 --2- 172.21.15.77:0/2700445132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d1c0d27b0 0x7f9d1c0d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.651+0000 7f9d13fff640 1 -- 172.21.15.77:0/2700445132 >> v1:172.21.15.121:6789/0 conn(0x7f9d1c0d23d0 legacy=0x7f9d1c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.651+0000 7f9d13fff640 1 -- 172.21.15.77:0/2700445132 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1c0cf2a0 con 0x7f9d1c0b9a90 2024-04-25T01:22:27.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.651+0000 7f9d13fff640 1 --2- 172.21.15.77:0/2700445132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d1c0b9a90 0x7f9d1c0d4920 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009870 tx=0x7f9d0402a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3407f6579d5cb476 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.651+0000 7f9d1a7fc640 1 -- 172.21.15.77:0/2700445132 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04037070 con 0x7f9d1c0b9a90 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.651+0000 7f9d1a7fc640 1 -- 172.21.15.77:0/2700445132 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d04032440 con 0x7f9d1c0b9a90 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.651+0000 7f9d1a7fc640 1 -- 172.21.15.77:0/2700445132 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04032810 con 0x7f9d1c0b9a90 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.652+0000 7f9d213e9640 1 -- 172.21.15.77:0/2700445132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d1c0b9a90 msgr2=0x7f9d1c0d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.652+0000 7f9d213e9640 1 --2- 172.21.15.77:0/2700445132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d1c0b9a90 0x7f9d1c0d4920 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009870 tx=0x7f9d0402a2a0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.652+0000 7f9d213e9640 1 -- 172.21.15.77:0/2700445132 shutdown_connections 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.652+0000 7f9d213e9640 1 --2- 172.21.15.77:0/2700445132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d1c0d27b0 0x7f9d1c0d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.652+0000 7f9d213e9640 1 --2- 172.21.15.77:0/2700445132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d1c0b9a90 0x7f9d1c0d4920 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.652+0000 7f9d213e9640 1 -- 172.21.15.77:0/2700445132 >> 172.21.15.77:0/2700445132 conn(0x7f9d1c0d3e30 msgr2=0x7f9d1c0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.652+0000 7f9d213e9640 1 -- 172.21.15.77:0/2700445132 shutdown_connections 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.652+0000 7f9d213e9640 1 -- 172.21.15.77:0/2700445132 wait complete. 2024-04-25T01:22:27.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.653+0000 7f9d213e9640 1 Processor -- start 2024-04-25T01:22:27.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.653+0000 7f9d213e9640 1 -- start start 2024-04-25T01:22:27.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d213e9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d1c0b9a90 0x7f9d1c14f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d213e9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d1c0d23d0 0x7f9d1c0ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d213e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d1c0d27b0 0x7f9d1c15e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d213e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c0cfd80 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d213e9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c0cf860 con 0x7f9d1c0d23d0 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d213e9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d1c0b6c80 con 0x7f9d1c0b9a90 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d1b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d1c0d27b0 0x7f9d1c15e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d1b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d1c0d27b0 0x7f9d1c15e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48890/0 (socket says 172.21.15.77:48890) 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d1b7fe640 1 -- 172.21.15.77:0/1028249126 learned_addr learned my addr 172.21.15.77:0/1028249126 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d13fff640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d1c0d23d0 0x7f9d1c0ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d1affd640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d1c0b9a90 0x7f9d1c14f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d1b7fe640 1 -- 172.21.15.77:0/1028249126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d1c0b9a90 msgr2=0x7f9d1c14f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d1b7fe640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d1c0b9a90 0x7f9d1c14f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d1b7fe640 1 -- 172.21.15.77:0/1028249126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d1c0d23d0 msgr2=0x7f9d1c0ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d1b7fe640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d1c0d23d0 0x7f9d1c0ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d1b7fe640 1 -- 172.21.15.77:0/1028249126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1c0c9290 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.654+0000 7f9d1b7fe640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d1c0d27b0 0x7f9d1c15e2d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9d14007e60 tx=0x7f9d1400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.655+0000 7f9d137fe640 1 -- 172.21.15.77:0/1028249126 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d14014420 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.655+0000 7f9d137fe640 1 -- 172.21.15.77:0/1028249126 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d14019070 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.655+0000 7f9d213e9640 1 -- 172.21.15.77:0/1028249126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1c0c8710 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.655+0000 7f9d137fe640 1 -- 172.21.15.77:0/1028249126 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d14014820 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.655+0000 7f9d213e9640 1 -- 172.21.15.77:0/1028249126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d1c0c8cd0 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.656+0000 7f9d213e9640 1 -- 172.21.15.77:0/1028249126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d1c068fb0 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.656+0000 7f9d137fe640 1 -- 172.21.15.77:0/1028249126 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9d140149c0 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.656+0000 7f9d137fe640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9cfc06be30 0x7f9cfc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.657+0000 7f9d137fe640 1 -- 172.21.15.77:0/1028249126 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f9d1408ba20 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.657+0000 7f9d1affd640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9cfc06be30 0x7f9cfc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.657+0000 7f9d1affd640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9cfc06be30 0x7f9cfc06e2f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c004ea0 tx=0x7f9d0c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.661+0000 7f9d137fe640 1 -- 172.21.15.77:0/1028249126 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9d1405a050 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.833+0000 7f9d213e9640 1 -- 172.21.15.77:0/1028249126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9d1c08df50 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.834+0000 7f9d137fe640 1 -- 172.21.15.77:0/1028249126 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f9d14059950 con 0x7f9d1c0d27b0 2024-04-25T01:22:27.815 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:27.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 -- 172.21.15.77:0/1028249126 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9cfc06be30 msgr2=0x7f9cfc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9cfc06be30 0x7f9cfc06e2f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c004ea0 tx=0x7f9d0c009040 comp rx=0 tx=0).stop 2024-04-25T01:22:27.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 -- 172.21.15.77:0/1028249126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d1c0d27b0 msgr2=0x7f9d1c15e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d1c0d27b0 0x7f9d1c15e2d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9d14007e60 tx=0x7f9d1400e4b0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 -- 172.21.15.77:0/1028249126 shutdown_connections 2024-04-25T01:22:27.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9cfc06be30 0x7f9cfc06e2f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d1c0d27b0 0x7f9d1c15e2d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d1c0d23d0 0x7f9d1c0ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 --2- 172.21.15.77:0/1028249126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d1c0b9a90 0x7f9d1c14f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 -- 172.21.15.77:0/1028249126 >> 172.21.15.77:0/1028249126 conn(0x7f9d1c0d3e30 msgr2=0x7f9d1c06c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:27.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 -- 172.21.15.77:0/1028249126 shutdown_connections 2024-04-25T01:22:27.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.836+0000 7f9d213e9640 1 -- 172.21.15.77:0/1028249126 wait complete. 2024-04-25T01:22:27.828 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:27.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.946+0000 7fd4162b3640 1 Processor -- start 2024-04-25T01:22:27.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.946+0000 7fd4162b3640 1 -- start start 2024-04-25T01:22:27.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd4162b3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4100af8c0 0x7fd4100cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd4162b3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd41006b830 0x7fd41015d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd4162b3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4100b0200 con 0x7fd4100b5060 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd4162b3640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fd4100b5840 con 0x7fd41006b830 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd4162b3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd4100b5b20 con 0x7fd4100af8c0 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd414850640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd41006b830 0x7fd41015d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd414850640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd41006b830 0x7fd41015d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48892/0 (socket says 172.21.15.77:48892) 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd414850640 1 -- 172.21.15.77:0/1701864954 learned_addr learned my addr 172.21.15.77:0/1701864954 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd414850640 1 -- 172.21.15.77:0/1701864954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4100af8c0 msgr2=0x7fd4100cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd414850640 1 --2- 172.21.15.77:0/1701864954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4100af8c0 0x7fd4100cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd414850640 1 -- 172.21.15.77:0/1701864954 >> v1:172.21.15.121:6789/0 conn(0x7fd4100b5060 legacy=0x7fd41006b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.947+0000 7fd414850640 1 -- 172.21.15.77:0/1701864954 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4100b5e00 con 0x7fd41006b830 2024-04-25T01:22:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.948+0000 7fd414850640 1 --2- 172.21.15.77:0/1701864954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd41006b830 0x7fd41015d940 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd408009e50 tx=0x7fd40802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aeb52bf135efba8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.948+0000 7fd40effd640 1 -- 172.21.15.77:0/1701864954 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd408036070 con 0x7fd41006b830 2024-04-25T01:22:27.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.948+0000 7fd40effd640 1 -- 172.21.15.77:0/1701864954 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd408031600 con 0x7fd41006b830 2024-04-25T01:22:27.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.948+0000 7fd40effd640 1 -- 172.21.15.77:0/1701864954 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4080319d0 con 0x7fd41006b830 2024-04-25T01:22:27.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.948+0000 7fd4162b3640 1 -- 172.21.15.77:0/1701864954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd41006b830 msgr2=0x7fd41015d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.948+0000 7fd4162b3640 1 --2- 172.21.15.77:0/1701864954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd41006b830 0x7fd41015d940 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd408009e50 tx=0x7fd40802a6c0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.949+0000 7fd4162b3640 1 -- 172.21.15.77:0/1701864954 shutdown_connections 2024-04-25T01:22:27.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.949+0000 7fd4162b3640 1 --2- 172.21.15.77:0/1701864954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd41006b830 0x7fd41015d940 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.949+0000 7fd4162b3640 1 --2- 172.21.15.77:0/1701864954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4100af8c0 0x7fd4100cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.949+0000 7fd4162b3640 1 -- 172.21.15.77:0/1701864954 >> 172.21.15.77:0/1701864954 conn(0x7fd4100b1f00 msgr2=0x7fd4100b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:27.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.949+0000 7fd4162b3640 1 -- 172.21.15.77:0/1701864954 shutdown_connections 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.949+0000 7fd4162b3640 1 -- 172.21.15.77:0/1701864954 wait complete. 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.950+0000 7fd4162b3640 1 Processor -- start 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.950+0000 7fd4162b3640 1 -- start start 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd4162b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd41006b830 0x7fd4100c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd4162b3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4100af8c0 0x7fd4100cf060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd4162b3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4100b5060 0x7fd4100cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd4162b3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd4100afca0 con 0x7fd41006b830 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd4162b3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd4100b5510 con 0x7fd4100af8c0 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd4162b3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd4100afe20 con 0x7fd4100b5060 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd41006b830 0x7fd4100c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd41006b830 0x7fd4100c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48906/0 (socket says 172.21.15.77:48906) 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40ffff640 1 -- 172.21.15.77:0/2837107000 learned_addr learned my addr 172.21.15.77:0/2837107000 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40ffff640 1 -- 172.21.15.77:0/2837107000 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4100b5060 msgr2=0x7fd4100cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40ffff640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4100b5060 0x7fd4100cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40ffff640 1 -- 172.21.15.77:0/2837107000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4100af8c0 msgr2=0x7fd4100cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40f7fe640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4100af8c0 0x7fd4100cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40ffff640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4100af8c0 0x7fd4100cf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40ffff640 1 -- 172.21.15.77:0/2837107000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4101e5f30 con 0x7fd41006b830 2024-04-25T01:22:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40f7fe640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4100af8c0 0x7fd4100cf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40ffff640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd41006b830 0x7fd4100c9a60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd40000b4a0 tx=0x7fd40000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40cff9640 1 -- 172.21.15.77:0/2837107000 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd400014070 con 0x7fd41006b830 2024-04-25T01:22:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40cff9640 1 -- 172.21.15.77:0/2837107000 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd400010610 con 0x7fd41006b830 2024-04-25T01:22:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd40cff9640 1 -- 172.21.15.77:0/2837107000 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd400010a00 con 0x7fd41006b830 2024-04-25T01:22:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd4162b3640 1 -- 172.21.15.77:0/2837107000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4101e6210 con 0x7fd41006b830 2024-04-25T01:22:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.951+0000 7fd4162b3640 1 -- 172.21.15.77:0/2837107000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4101e67d0 con 0x7fd41006b830 2024-04-25T01:22:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.952+0000 7fd4162b3640 1 -- 172.21.15.77:0/2837107000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd410068fb0 con 0x7fd41006b830 2024-04-25T01:22:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.953+0000 7fd40cff9640 1 -- 172.21.15.77:0/2837107000 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd400017070 con 0x7fd41006b830 2024-04-25T01:22:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.953+0000 7fd40cff9640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd3fc06be30 0x7fd3fc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.953+0000 7fd40f7fe640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd3fc06be30 0x7fd3fc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.953+0000 7fd40cff9640 1 -- 172.21.15.77:0/2837107000 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fd40008ad00 con 0x7fd41006b830 2024-04-25T01:22:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.954+0000 7fd40f7fe640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd3fc06be30 0x7fd3fc06e2f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd41006d9e0 tx=0x7fd404008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:27.957+0000 7fd40cff9640 1 -- 172.21.15.77:0/2837107000 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd4000592a0 con 0x7fd41006b830 2024-04-25T01:22:28.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.128+0000 7fd4162b3640 1 -- 172.21.15.77:0/2837107000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd4100b16f0 con 0x7fd41006b830 2024-04-25T01:22:28.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.128+0000 7fd40cff9640 1 -- 172.21.15.77:0/2837107000 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fd400058ba0 con 0x7fd41006b830 2024-04-25T01:22:28.110 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:28.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 -- 172.21.15.77:0/2837107000 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd3fc06be30 msgr2=0x7fd3fc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd3fc06be30 0x7fd3fc06e2f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd41006d9e0 tx=0x7fd404008040 comp rx=0 tx=0).stop 2024-04-25T01:22:28.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 -- 172.21.15.77:0/2837107000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd41006b830 msgr2=0x7fd4100c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd41006b830 0x7fd4100c9a60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd40000b4a0 tx=0x7fd40000b960 comp rx=0 tx=0).stop 2024-04-25T01:22:28.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 -- 172.21.15.77:0/2837107000 shutdown_connections 2024-04-25T01:22:28.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd3fc06be30 0x7fd3fc06e2f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4100b5060 0x7fd4100cf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4100af8c0 0x7fd4100cf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 --2- 172.21.15.77:0/2837107000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd41006b830 0x7fd4100c9a60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 -- 172.21.15.77:0/2837107000 >> 172.21.15.77:0/2837107000 conn(0x7fd4100b1f00 msgr2=0x7fd4100cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:28.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 -- 172.21.15.77:0/2837107000 shutdown_connections 2024-04-25T01:22:28.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.130+0000 7fd4162b3640 1 -- 172.21.15.77:0/2837107000 wait complete. 2024-04-25T01:22:28.123 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:28.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.242+0000 7f1906266640 1 Processor -- start 2024-04-25T01:22:28.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.242+0000 7f1906266640 1 -- start start 2024-04-25T01:22:28.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.243+0000 7f1906266640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f190006caf0 0x7f19000b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.243+0000 7f1906266640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f190006da80 0x7f1900070b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.243+0000 7f1906266640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f190015bb80 con 0x7f19000720c0 2024-04-25T01:22:28.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.243+0000 7f1906266640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f19000724a0 con 0x7f190006caf0 2024-04-25T01:22:28.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.243+0000 7f1906266640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f19000b3750 con 0x7f190006da80 2024-04-25T01:22:28.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.243+0000 7f18fffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f19000720c0 0x7f190006d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57398/0 (socket says 172.21.15.77:57398) 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.243+0000 7f18fffff640 1 -- 172.21.15.77:0/1576951500 learned_addr learned my addr 172.21.15.77:0/1576951500 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.243+0000 7f18ff7fe640 1 --2- 172.21.15.77:0/1576951500 >> v2:172.21.15.77:3300/0 conn(0x7f190006caf0 0x7f19000b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.244+0000 7f1904803640 1 --2- 172.21.15.77:0/1576951500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f190006da80 0x7f1900070b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.244+0000 7f18feffd640 1 -- 172.21.15.77:0/1576951500 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 647508972 0 0) 0x7f190015bb80 con 0x7f19000720c0 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.244+0000 7f18feffd640 1 -- 172.21.15.77:0/1576951500 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18e8003440 con 0x7f19000720c0 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.244+0000 7f18ff7fe640 1 -- 172.21.15.77:0/1576951500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f190006da80 msgr2=0x7f1900070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.244+0000 7f18ff7fe640 1 --2- 172.21.15.77:0/1576951500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f190006da80 0x7f1900070b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.244+0000 7f18ff7fe640 1 -- 172.21.15.77:0/1576951500 >> v1:172.21.15.77:6789/0 conn(0x7f19000720c0 legacy=0x7f190006d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.244+0000 7f18ff7fe640 1 -- 172.21.15.77:0/1576951500 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19000b3a30 con 0x7f190006caf0 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.244+0000 7f1904803640 1 --2- 172.21.15.77:0/1576951500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f190006da80 0x7f1900070b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.244+0000 7f18ff7fe640 1 --2- 172.21.15.77:0/1576951500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f190006caf0 0x7f19000b4510 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f18f40097a0 tx=0x7f18f402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef6dd0f667d95a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.244+0000 7f18feffd640 1 -- 172.21.15.77:0/1576951500 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4037070 con 0x7f190006caf0 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.245+0000 7f18feffd640 1 -- 172.21.15.77:0/1576951500 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18f402ae50 con 0x7f190006caf0 2024-04-25T01:22:28.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.245+0000 7f18feffd640 1 -- 172.21.15.77:0/1576951500 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4032660 con 0x7f190006caf0 2024-04-25T01:22:28.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.245+0000 7f1906266640 1 -- 172.21.15.77:0/1576951500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f190006caf0 msgr2=0x7f19000b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.245+0000 7f1906266640 1 --2- 172.21.15.77:0/1576951500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f190006caf0 0x7f19000b4510 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f18f40097a0 tx=0x7f18f402a100 comp rx=0 tx=0).stop 2024-04-25T01:22:28.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.245+0000 7f1906266640 1 -- 172.21.15.77:0/1576951500 shutdown_connections 2024-04-25T01:22:28.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.245+0000 7f1906266640 1 --2- 172.21.15.77:0/1576951500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f190006da80 0x7f1900070b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.245+0000 7f1906266640 1 --2- 172.21.15.77:0/1576951500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f190006caf0 0x7f19000b4510 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.245+0000 7f1906266640 1 -- 172.21.15.77:0/1576951500 >> 172.21.15.77:0/1576951500 conn(0x7f190006b720 msgr2=0x7f19000b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:28.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.245+0000 7f1906266640 1 -- 172.21.15.77:0/1576951500 shutdown_connections 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.245+0000 7f1906266640 1 -- 172.21.15.77:0/1576951500 wait complete. 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.246+0000 7f1906266640 1 Processor -- start 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.246+0000 7f1906266640 1 -- start start 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.247+0000 7f1906266640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f190006caf0 0x7f1900145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.247+0000 7f1906266640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f190006da80 0x7f19001462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.247+0000 7f1906266640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19000720c0 0x7f1900159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.247+0000 7f1906266640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f190015b6a0 con 0x7f190006caf0 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.247+0000 7f1906266640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f19000b3ff0 con 0x7f19000720c0 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.247+0000 7f1906266640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f190015b820 con 0x7f190006da80 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.247+0000 7f18fffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f190006caf0 0x7f1900145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.247+0000 7f18fffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f190006caf0 0x7f1900145d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48924/0 (socket says 172.21.15.77:48924) 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.247+0000 7f18fffff640 1 -- 172.21.15.77:0/1572452592 learned_addr learned my addr 172.21.15.77:0/1572452592 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.247+0000 7f18fffff640 1 -- 172.21.15.77:0/1572452592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f190006da80 msgr2=0x7f19001462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.247+0000 7f18fffff640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f190006da80 0x7f19001462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.248+0000 7f18fffff640 1 -- 172.21.15.77:0/1572452592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19000720c0 msgr2=0x7f1900159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.248+0000 7f1904803640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19000720c0 0x7f1900159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.248+0000 7f18fffff640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19000720c0 0x7f1900159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.248+0000 7f18fffff640 1 -- 172.21.15.77:0/1572452592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19000cd830 con 0x7f190006caf0 2024-04-25T01:22:28.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.248+0000 7f18fffff640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f190006caf0 0x7f1900145d70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f18f000c390 tx=0x7f18f000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:28.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.248+0000 7f18fcff9640 1 -- 172.21.15.77:0/1572452592 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f0017070 con 0x7f190006caf0 2024-04-25T01:22:28.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.248+0000 7f18fcff9640 1 -- 172.21.15.77:0/1572452592 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18f0005ce0 con 0x7f190006caf0 2024-04-25T01:22:28.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.248+0000 7f1906266640 1 -- 172.21.15.77:0/1572452592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19000cdb10 con 0x7f190006caf0 2024-04-25T01:22:28.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.248+0000 7f18fcff9640 1 -- 172.21.15.77:0/1572452592 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f00072f0 con 0x7f190006caf0 2024-04-25T01:22:28.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.248+0000 7f1906266640 1 -- 172.21.15.77:0/1572452592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19000ce0d0 con 0x7f190006caf0 2024-04-25T01:22:28.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.249+0000 7f1906266640 1 -- 172.21.15.77:0/1572452592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1900068fb0 con 0x7f190006caf0 2024-04-25T01:22:28.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.249+0000 7f18fcff9640 1 -- 172.21.15.77:0/1572452592 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f18f001a070 con 0x7f190006caf0 2024-04-25T01:22:28.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.250+0000 7f18fcff9640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f18ec06be30 0x7f18ec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.250+0000 7f18ff7fe640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f18ec06be30 0x7f18ec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.250+0000 7f18fcff9640 1 -- 172.21.15.77:0/1572452592 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f18f008a640 con 0x7f190006caf0 2024-04-25T01:22:28.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.250+0000 7f18ff7fe640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f18ec06be30 0x7f18ec06e2f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f18f4034000 tx=0x7f18f402c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:28.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.254+0000 7f18fcff9640 1 -- 172.21.15.77:0/1572452592 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f18f0058ca0 con 0x7f190006caf0 2024-04-25T01:22:28.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.429+0000 7f1906266640 1 -- 172.21.15.77:0/1572452592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f19000b3400 con 0x7f190006caf0 2024-04-25T01:22:28.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.429+0000 7f18fcff9640 1 -- 172.21.15.77:0/1572452592 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f18f00585a0 con 0x7f190006caf0 2024-04-25T01:22:28.411 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 -- 172.21.15.77:0/1572452592 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f18ec06be30 msgr2=0x7f18ec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f18ec06be30 0x7f18ec06e2f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f18f4034000 tx=0x7f18f402c000 comp rx=0 tx=0).stop 2024-04-25T01:22:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 -- 172.21.15.77:0/1572452592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f190006caf0 msgr2=0x7f1900145d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f190006caf0 0x7f1900145d70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f18f000c390 tx=0x7f18f000c850 comp rx=0 tx=0).stop 2024-04-25T01:22:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 -- 172.21.15.77:0/1572452592 shutdown_connections 2024-04-25T01:22:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f18ec06be30 0x7f18ec06e2f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19000720c0 0x7f1900159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f190006da80 0x7f19001462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 --2- 172.21.15.77:0/1572452592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f190006caf0 0x7f1900145d70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 -- 172.21.15.77:0/1572452592 >> 172.21.15.77:0/1572452592 conn(0x7f190006b720 msgr2=0x7f190015dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 -- 172.21.15.77:0/1572452592 shutdown_connections 2024-04-25T01:22:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.431+0000 7f1906266640 1 -- 172.21.15.77:0/1572452592 wait complete. 2024-04-25T01:22:28.424 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:28.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.541+0000 7fb3c822c640 1 Processor -- start 2024-04-25T01:22:28.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.542+0000 7fb3c822c640 1 -- start start 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.542+0000 7fb3c822c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3c0158a70 0x7fb3c00b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.542+0000 7fb3c822c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3c015bc30 0x7fb3c00b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.542+0000 7fb3c822c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3c015c210 con 0x7fb3c00b37c0 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.542+0000 7fb3c822c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fb3c00b2ce0 con 0x7fb3c015bc30 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.542+0000 7fb3c822c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb3c00b2fc0 con 0x7fb3c0158a70 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.543+0000 7fb3c67c9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3c015bc30 0x7fb3c00b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.543+0000 7fb3c67c9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3c015bc30 0x7fb3c00b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48926/0 (socket says 172.21.15.77:48926) 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.543+0000 7fb3c67c9640 1 -- 172.21.15.77:0/125019930 learned_addr learned my addr 172.21.15.77:0/125019930 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.543+0000 7fb3c5fc8640 1 --2- 172.21.15.77:0/125019930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3c0158a70 0x7fb3c00b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.543+0000 7fb3c67c9640 1 -- 172.21.15.77:0/125019930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3c0158a70 msgr2=0x7fb3c00b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.543+0000 7fb3c67c9640 1 --2- 172.21.15.77:0/125019930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3c0158a70 0x7fb3c00b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.543+0000 7fb3c67c9640 1 -- 172.21.15.77:0/125019930 >> v1:172.21.15.121:6789/0 conn(0x7fb3c00b37c0 legacy=0x7fb3c015b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:28.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.543+0000 7fb3c67c9640 1 -- 172.21.15.77:0/125019930 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c00b32a0 con 0x7fb3c015bc30 2024-04-25T01:22:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c67c9640 1 --2- 172.21.15.77:0/125019930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c015bc30 0x7fb3c00b4600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc009870 tx=0x7fb3bc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8946bdd3e198b3a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c4fc6640 1 -- 172.21.15.77:0/125019930 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc037070 con 0x7fb3c015bc30 2024-04-25T01:22:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c4fc6640 1 -- 172.21.15.77:0/125019930 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3bc032440 con 0x7fb3c015bc30 2024-04-25T01:22:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c4fc6640 1 -- 172.21.15.77:0/125019930 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc032810 con 0x7fb3c015bc30 2024-04-25T01:22:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c822c640 1 -- 172.21.15.77:0/125019930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c015bc30 msgr2=0x7fb3c00b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c822c640 1 --2- 172.21.15.77:0/125019930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c015bc30 0x7fb3c00b4600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc009870 tx=0x7fb3bc02a210 comp rx=0 tx=0).stop 2024-04-25T01:22:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c822c640 1 -- 172.21.15.77:0/125019930 shutdown_connections 2024-04-25T01:22:28.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c822c640 1 --2- 172.21.15.77:0/125019930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c015bc30 0x7fb3c00b4600 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c822c640 1 --2- 172.21.15.77:0/125019930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3c0158a70 0x7fb3c00b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c822c640 1 -- 172.21.15.77:0/125019930 >> 172.21.15.77:0/125019930 conn(0x7fb3c006b720 msgr2=0x7fb3c00b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:28.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c822c640 1 -- 172.21.15.77:0/125019930 shutdown_connections 2024-04-25T01:22:28.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.544+0000 7fb3c822c640 1 -- 172.21.15.77:0/125019930 wait complete. 2024-04-25T01:22:28.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.545+0000 7fb3c822c640 1 Processor -- start 2024-04-25T01:22:28.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c822c640 1 -- start start 2024-04-25T01:22:28.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c822c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3c00b37c0 0x7fb3c015c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c822c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0158a70 0x7fb3c0167870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c822c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3c015bc30 0x7fb3c00cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c822c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c006cb70 con 0x7fb3c0158a70 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c822c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c006c9f0 con 0x7fb3c015bc30 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c822c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb3c006ccf0 con 0x7fb3c00b37c0 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c57c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0158a70 0x7fb3c0167870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c57c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0158a70 0x7fb3c0167870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48932/0 (socket says 172.21.15.77:48932) 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c57c7640 1 -- 172.21.15.77:0/136851824 learned_addr learned my addr 172.21.15.77:0/136851824 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c5fc8640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3c00b37c0 0x7fb3c015c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.546+0000 7fb3c67c9640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3c015bc30 0x7fb3c00cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3c57c7640 1 -- 172.21.15.77:0/136851824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3c00b37c0 msgr2=0x7fb3c015c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3c57c7640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3c00b37c0 0x7fb3c015c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3c57c7640 1 -- 172.21.15.77:0/136851824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3c015bc30 msgr2=0x7fb3c00cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3c57c7640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3c015bc30 0x7fb3c00cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3c57c7640 1 -- 172.21.15.77:0/136851824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c015cbd0 con 0x7fb3c0158a70 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3c5fc8640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3c00b37c0 0x7fb3c015c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3c67c9640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3c015bc30 0x7fb3c00cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3c57c7640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0158a70 0x7fb3c0167870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb3b000c430 tx=0x7fb3b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3b67fc640 1 -- 172.21.15.77:0/136851824 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0011790 con 0x7fb3c0158a70 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3b67fc640 1 -- 172.21.15.77:0/136851824 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3b0017070 con 0x7fb3c0158a70 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3c822c640 1 -- 172.21.15.77:0/136851824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3c015ceb0 con 0x7fb3c0158a70 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3b67fc640 1 -- 172.21.15.77:0/136851824 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0011b70 con 0x7fb3c0158a70 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.547+0000 7fb3c822c640 1 -- 172.21.15.77:0/136851824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3c015d470 con 0x7fb3c0158a70 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.548+0000 7fb3c822c640 1 -- 172.21.15.77:0/136851824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3c0068fb0 con 0x7fb3c0158a70 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.548+0000 7fb3b67fc640 1 -- 172.21.15.77:0/136851824 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb3b0011d10 con 0x7fb3c0158a70 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.549+0000 7fb3b67fc640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb3a406be30 0x7fb3a406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.549+0000 7fb3b67fc640 1 -- 172.21.15.77:0/136851824 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fb3b008b090 con 0x7fb3c0158a70 2024-04-25T01:22:28.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.549+0000 7fb3c5fc8640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb3a406be30 0x7fb3a406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.549+0000 7fb3c5fc8640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb3a406be30 0x7fb3a406e2f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8009b20 tx=0x7fb3b80039d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:28.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.553+0000 7fb3b67fc640 1 -- 172.21.15.77:0/136851824 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb3b00596c0 con 0x7fb3c0158a70 2024-04-25T01:22:28.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.725+0000 7fb3c822c640 1 -- 172.21.15.77:0/136851824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb3c013cb30 con 0x7fb3c0158a70 2024-04-25T01:22:28.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.725+0000 7fb3b67fc640 1 -- 172.21.15.77:0/136851824 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fb3b0058fc0 con 0x7fb3c0158a70 2024-04-25T01:22:28.707 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:28.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 -- 172.21.15.77:0/136851824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb3a406be30 msgr2=0x7fb3a406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb3a406be30 0x7fb3a406e2f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8009b20 tx=0x7fb3b80039d0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 -- 172.21.15.77:0/136851824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0158a70 msgr2=0x7fb3c0167870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0158a70 0x7fb3c0167870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb3b000c430 tx=0x7fb3b000c8f0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 -- 172.21.15.77:0/136851824 shutdown_connections 2024-04-25T01:22:28.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb3a406be30 0x7fb3a406e2f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0158a70 0x7fb3c0167870 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3c015bc30 0x7fb3c00cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 --2- 172.21.15.77:0/136851824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3c00b37c0 0x7fb3c015c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 -- 172.21.15.77:0/136851824 >> 172.21.15.77:0/136851824 conn(0x7fb3c006b720 msgr2=0x7fb3c013bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:28.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 -- 172.21.15.77:0/136851824 shutdown_connections 2024-04-25T01:22:28.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.727+0000 7fb3c822c640 1 -- 172.21.15.77:0/136851824 wait complete. 2024-04-25T01:22:28.720 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:28.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.835+0000 7f8b65231640 1 Processor -- start 2024-04-25T01:22:28.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.835+0000 7f8b65231640 1 -- start start 2024-04-25T01:22:28.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b65231640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8b600b4db0 0x7f8b60162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b65231640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b6015e1f0 0x7f8b600719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b65231640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b600b3a10 con 0x7f8b6015de10 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b65231640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8b600b3cf0 con 0x7f8b600b4db0 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b65231640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8b600b3fd0 con 0x7f8b6015e1f0 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b5f59e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b6015e1f0 0x7f8b600719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b57fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8b600b4db0 0x7f8b60162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b5f59e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b6015e1f0 0x7f8b600719c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:55908/0 (socket says 172.21.15.77:55908) 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b5f59e640 1 -- 172.21.15.77:0/1801065558 learned_addr learned my addr 172.21.15.77:0/1801065558 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b5f59e640 1 -- 172.21.15.77:0/1801065558 >> v2:172.21.15.121:3300/0 conn(0x7f8b600b4db0 msgr2=0x7f8b60162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b5f59e640 1 --2- 172.21.15.77:0/1801065558 >> v2:172.21.15.121:3300/0 conn(0x7f8b600b4db0 0x7f8b60162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b5f59e640 1 -- 172.21.15.77:0/1801065558 >> v1:172.21.15.121:6789/0 conn(0x7f8b6015de10 legacy=0x7f8b60161c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.836+0000 7f8b5f59e640 1 -- 172.21.15.77:0/1801065558 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b600b4660 con 0x7f8b6015e1f0 2024-04-25T01:22:28.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.837+0000 7f8b5f59e640 1 --2- 172.21.15.77:0/1801065558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b6015e1f0 0x7f8b600719c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009e50 tx=0x7f8b5802a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f9846edee52a9a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:28.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.837+0000 7f8b5e59c640 1 -- 172.21.15.77:0/1801065558 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58036070 con 0x7f8b6015e1f0 2024-04-25T01:22:28.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.837+0000 7f8b5e59c640 1 -- 172.21.15.77:0/1801065558 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b58031600 con 0x7f8b6015e1f0 2024-04-25T01:22:28.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.837+0000 7f8b5e59c640 1 -- 172.21.15.77:0/1801065558 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b580319d0 con 0x7f8b6015e1f0 2024-04-25T01:22:28.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.838+0000 7f8b65231640 1 -- 172.21.15.77:0/1801065558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b6015e1f0 msgr2=0x7f8b600719c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.838+0000 7f8b65231640 1 --2- 172.21.15.77:0/1801065558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b6015e1f0 0x7f8b600719c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009e50 tx=0x7f8b5802a5f0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.838+0000 7f8b65231640 1 -- 172.21.15.77:0/1801065558 shutdown_connections 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.838+0000 7f8b65231640 1 --2- 172.21.15.77:0/1801065558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b6015e1f0 0x7f8b600719c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.838+0000 7f8b65231640 1 --2- 172.21.15.77:0/1801065558 >> v2:172.21.15.121:3300/0 conn(0x7f8b600b4db0 0x7f8b60162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.838+0000 7f8b65231640 1 -- 172.21.15.77:0/1801065558 >> 172.21.15.77:0/1801065558 conn(0x7f8b600b1820 msgr2=0x7f8b600b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.838+0000 7f8b65231640 1 -- 172.21.15.77:0/1801065558 shutdown_connections 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.838+0000 7f8b65231640 1 -- 172.21.15.77:0/1801065558 wait complete. 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.839+0000 7f8b65231640 1 Processor -- start 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.839+0000 7f8b65231640 1 -- start start 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b65231640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b600b4db0 0x7f8b60152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b65231640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b6015de10 0x7f8b601ea5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b65231640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b6015e1f0 0x7f8b601e8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b65231640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8b60072080 con 0x7f8b6015e1f0 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b65231640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8b60071f00 con 0x7f8b6015de10 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b65231640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8b60072200 con 0x7f8b600b4db0 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b5f59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b6015e1f0 0x7f8b601e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b5f59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b6015e1f0 0x7f8b601e8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48944/0 (socket says 172.21.15.77:48944) 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b5f59e640 1 -- 172.21.15.77:0/1580025398 learned_addr learned my addr 172.21.15.77:0/1580025398 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b57fff640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b6015de10 0x7f8b601ea5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b5ed9d640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b600b4db0 0x7f8b60152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b5f59e640 1 -- 172.21.15.77:0/1580025398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b600b4db0 msgr2=0x7f8b60152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b5f59e640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b600b4db0 0x7f8b60152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b5f59e640 1 -- 172.21.15.77:0/1580025398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b6015de10 msgr2=0x7f8b601ea5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b5f59e640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b6015de10 0x7f8b601ea5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b5f59e640 1 -- 172.21.15.77:0/1580025398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b60153100 con 0x7f8b6015e1f0 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b5ed9d640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b600b4db0 0x7f8b60152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:28.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b57fff640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b6015de10 0x7f8b601ea5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:28.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b5f59e640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b6015e1f0 0x7f8b601e8360 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009e50 tx=0x7f8b58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:28.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.840+0000 7f8b577fe640 1 -- 172.21.15.77:0/1580025398 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58036050 con 0x7f8b6015e1f0 2024-04-25T01:22:28.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.841+0000 7f8b577fe640 1 -- 172.21.15.77:0/1580025398 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b58042070 con 0x7f8b6015e1f0 2024-04-25T01:22:28.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.841+0000 7f8b577fe640 1 -- 172.21.15.77:0/1580025398 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58009510 con 0x7f8b6015e1f0 2024-04-25T01:22:28.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.841+0000 7f8b65231640 1 -- 172.21.15.77:0/1580025398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b601533e0 con 0x7f8b6015e1f0 2024-04-25T01:22:28.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.841+0000 7f8b65231640 1 -- 172.21.15.77:0/1580025398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b601539a0 con 0x7f8b6015e1f0 2024-04-25T01:22:28.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.841+0000 7f8b65231640 1 -- 172.21.15.77:0/1580025398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b60068fb0 con 0x7f8b6015e1f0 2024-04-25T01:22:28.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.842+0000 7f8b577fe640 1 -- 172.21.15.77:0/1580025398 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8b58007bf0 con 0x7f8b6015e1f0 2024-04-25T01:22:28.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.842+0000 7f8b577fe640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b4006be30 0x7f8b4006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.843+0000 7f8b5ed9d640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b4006be30 0x7f8b4006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.843+0000 7f8b577fe640 1 -- 172.21.15.77:0/1580025398 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f8b580ac020 con 0x7f8b6015e1f0 2024-04-25T01:22:28.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.843+0000 7f8b5ed9d640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b4006be30 0x7f8b4006e2f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8b50001700 tx=0x7f8b50009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:28.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:28.847+0000 7f8b577fe640 1 -- 172.21.15.77:0/1580025398 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8b5807a6f0 con 0x7f8b6015e1f0 2024-04-25T01:22:29.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.018+0000 7f8b65231640 1 -- 172.21.15.77:0/1580025398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b600af210 con 0x7f8b6015e1f0 2024-04-25T01:22:29.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.020+0000 7f8b577fe640 1 -- 172.21.15.77:0/1580025398 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18922 (secure 0 0 0) 0x7f8b58079ff0 con 0x7f8b6015e1f0 2024-04-25T01:22:29.002 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:29.002 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:26.287128+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:29.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 -- 172.21.15.77:0/1580025398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b4006be30 msgr2=0x7f8b4006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b4006be30 0x7f8b4006e2f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8b50001700 tx=0x7f8b50009040 comp rx=0 tx=0).stop 2024-04-25T01:22:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 -- 172.21.15.77:0/1580025398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b6015e1f0 msgr2=0x7f8b601e8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b6015e1f0 0x7f8b601e8360 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009e50 tx=0x7f8b58004a40 comp rx=0 tx=0).stop 2024-04-25T01:22:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 -- 172.21.15.77:0/1580025398 shutdown_connections 2024-04-25T01:22:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b4006be30 0x7f8b4006e2f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b6015e1f0 0x7f8b601e8360 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b6015de10 0x7f8b601ea5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 --2- 172.21.15.77:0/1580025398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b600b4db0 0x7f8b60152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 -- 172.21.15.77:0/1580025398 >> 172.21.15.77:0/1580025398 conn(0x7f8b600b1820 msgr2=0x7f8b600affc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 -- 172.21.15.77:0/1580025398 shutdown_connections 2024-04-25T01:22:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.022+0000 7f8b65231640 1 -- 172.21.15.77:0/1580025398 wait complete. 2024-04-25T01:22:29.015 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:29.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:29.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.135+0000 7fb87eea5640 1 Processor -- start 2024-04-25T01:22:29.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.135+0000 7fb87eea5640 1 -- start start 2024-04-25T01:22:29.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87eea5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb87815fc40 0x7fb8781600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87eea5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb8780cedc0 0x7fb8780cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87eea5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8780cf790 con 0x7fb8780722e0 2024-04-25T01:22:29.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87eea5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fb878160aa0 con 0x7fb8780cedc0 2024-04-25T01:22:29.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87eea5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb87806da40 con 0x7fb87815fc40 2024-04-25T01:22:29.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87cc41640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb8780722e0 0x7fb87806ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57402/0 (socket says 172.21.15.77:57402) 2024-04-25T01:22:29.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87cc41640 1 -- 172.21.15.77:0/2094215565 learned_addr learned my addr 172.21.15.77:0/2094215565 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:29.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87d442640 1 --2- 172.21.15.77:0/2094215565 >> v2:172.21.15.77:3300/0 conn(0x7fb8780cedc0 0x7fb8780cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb86ffff640 1 --2- 172.21.15.77:0/2094215565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb87815fc40 0x7fb8781600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb86f7fe640 1 -- 172.21.15.77:0/2094215565 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2660620947 0 0) 0x7fb874001110 con 0x7fb8780722e0 2024-04-25T01:22:29.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb86f7fe640 1 -- 172.21.15.77:0/2094215565 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb860003520 con 0x7fb8780722e0 2024-04-25T01:22:29.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87d442640 1 -- 172.21.15.77:0/2094215565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb87815fc40 msgr2=0x7fb8781600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87d442640 1 --2- 172.21.15.77:0/2094215565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb87815fc40 0x7fb8781600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87d442640 1 -- 172.21.15.77:0/2094215565 >> v1:172.21.15.77:6789/0 conn(0x7fb8780722e0 legacy=0x7fb87806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.136+0000 7fb87d442640 1 -- 172.21.15.77:0/2094215565 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87806d110 con 0x7fb8780cedc0 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.137+0000 7fb87d442640 1 --2- 172.21.15.77:0/2094215565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780cedc0 0x7fb8780cf250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb870009d90 tx=0x7fb87002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b03728e929f6ca5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.137+0000 7fb86f7fe640 1 -- 172.21.15.77:0/2094215565 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb870031600 con 0x7fb8780cedc0 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.137+0000 7fb86f7fe640 1 -- 172.21.15.77:0/2094215565 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8700317a0 con 0x7fb8780cedc0 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.137+0000 7fb86f7fe640 1 -- 172.21.15.77:0/2094215565 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb870031b70 con 0x7fb8780cedc0 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.138+0000 7fb87eea5640 1 -- 172.21.15.77:0/2094215565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780cedc0 msgr2=0x7fb8780cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.138+0000 7fb87eea5640 1 --2- 172.21.15.77:0/2094215565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780cedc0 0x7fb8780cf250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb870009d90 tx=0x7fb87002a5d0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.138+0000 7fb87eea5640 1 -- 172.21.15.77:0/2094215565 shutdown_connections 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.138+0000 7fb87eea5640 1 --2- 172.21.15.77:0/2094215565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780cedc0 0x7fb8780cf250 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.138+0000 7fb87eea5640 1 --2- 172.21.15.77:0/2094215565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb87815fc40 0x7fb8781600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.138+0000 7fb87eea5640 1 -- 172.21.15.77:0/2094215565 >> 172.21.15.77:0/2094215565 conn(0x7fb87806b720 msgr2=0x7fb8780b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:29.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.138+0000 7fb87eea5640 1 -- 172.21.15.77:0/2094215565 shutdown_connections 2024-04-25T01:22:29.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.138+0000 7fb87eea5640 1 -- 172.21.15.77:0/2094215565 wait complete. 2024-04-25T01:22:29.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.139+0000 7fb87eea5640 1 Processor -- start 2024-04-25T01:22:29.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.139+0000 7fb87eea5640 1 -- start start 2024-04-25T01:22:29.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87eea5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8780722e0 0x7fb878155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87eea5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780cedc0 0x7fb878136b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87eea5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb87815fc40 0x7fb8781370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87eea5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb87806cec0 con 0x7fb8780cedc0 2024-04-25T01:22:29.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87eea5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb87806d6d0 con 0x7fb8780722e0 2024-04-25T01:22:29.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87eea5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb878070b50 con 0x7fb87815fc40 2024-04-25T01:22:29.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780cedc0 0x7fb878136b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780cedc0 0x7fb878136b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48970/0 (socket says 172.21.15.77:48970) 2024-04-25T01:22:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86ffff640 1 -- 172.21.15.77:0/3110503998 learned_addr learned my addr 172.21.15.77:0/3110503998 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87d442640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb87815fc40 0x7fb8781370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87cc41640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8780722e0 0x7fb878155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86ffff640 1 -- 172.21.15.77:0/3110503998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb87815fc40 msgr2=0x7fb8781370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86ffff640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb87815fc40 0x7fb8781370c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86ffff640 1 -- 172.21.15.77:0/3110503998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8780722e0 msgr2=0x7fb878155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86ffff640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8780722e0 0x7fb878155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86ffff640 1 -- 172.21.15.77:0/3110503998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb878157570 con 0x7fb8780cedc0 2024-04-25T01:22:29.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87d442640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb87815fc40 0x7fb8781370c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:29.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87cc41640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8780722e0 0x7fb878155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:29.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86ffff640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780cedc0 0x7fb878136b80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb86800e870 tx=0x7fb86800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:29.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86d7fa640 1 -- 172.21.15.77:0/3110503998 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868017070 con 0x7fb8780cedc0 2024-04-25T01:22:29.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86d7fa640 1 -- 172.21.15.77:0/3110503998 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb86800c940 con 0x7fb8780cedc0 2024-04-25T01:22:29.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb86d7fa640 1 -- 172.21.15.77:0/3110503998 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86800cd30 con 0x7fb8780cedc0 2024-04-25T01:22:29.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87eea5640 1 -- 172.21.15.77:0/3110503998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8780a9890 con 0x7fb8780cedc0 2024-04-25T01:22:29.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.140+0000 7fb87eea5640 1 -- 172.21.15.77:0/3110503998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb878157830 con 0x7fb8780cedc0 2024-04-25T01:22:29.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.141+0000 7fb87eea5640 1 -- 172.21.15.77:0/3110503998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb878068fb0 con 0x7fb8780cedc0 2024-04-25T01:22:29.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.142+0000 7fb86d7fa640 1 -- 172.21.15.77:0/3110503998 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb86801a070 con 0x7fb8780cedc0 2024-04-25T01:22:29.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.142+0000 7fb86d7fa640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb86406be30 0x7fb86406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.142+0000 7fb87cc41640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb86406be30 0x7fb86406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.142+0000 7fb86d7fa640 1 -- 172.21.15.77:0/3110503998 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fb86808b090 con 0x7fb8780cedc0 2024-04-25T01:22:29.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.143+0000 7fb87cc41640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb86406be30 0x7fb86406e2f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb8740012b0 tx=0x7fb874009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:29.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.147+0000 7fb86d7fa640 1 -- 172.21.15.77:0/3110503998 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb8680596c0 con 0x7fb8780cedc0 2024-04-25T01:22:29.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.325+0000 7fb87eea5640 1 -- 172.21.15.77:0/3110503998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb878072c90 con 0x7fb8780cedc0 2024-04-25T01:22:29.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.325+0000 7fb86d7fa640 1 -- 172.21.15.77:0/3110503998 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fb868058fc0 con 0x7fb8780cedc0 2024-04-25T01:22:29.307 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:29.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.327+0000 7fb87eea5640 1 -- 172.21.15.77:0/3110503998 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb86406be30 msgr2=0x7fb86406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.327+0000 7fb87eea5640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb86406be30 0x7fb86406e2f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb8740012b0 tx=0x7fb874009040 comp rx=0 tx=0).stop 2024-04-25T01:22:29.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.327+0000 7fb87eea5640 1 -- 172.21.15.77:0/3110503998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780cedc0 msgr2=0x7fb878136b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.327+0000 7fb87eea5640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780cedc0 0x7fb878136b80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb86800e870 tx=0x7fb86800ed30 comp rx=0 tx=0).stop 2024-04-25T01:22:29.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.327+0000 7fb87eea5640 1 -- 172.21.15.77:0/3110503998 shutdown_connections 2024-04-25T01:22:29.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.327+0000 7fb87eea5640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb86406be30 0x7fb86406e2f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.327+0000 7fb87eea5640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780cedc0 0x7fb878136b80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.327+0000 7fb87eea5640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb87815fc40 0x7fb8781370c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.327+0000 7fb87eea5640 1 --2- 172.21.15.77:0/3110503998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8780722e0 0x7fb878155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.327+0000 7fb87eea5640 1 -- 172.21.15.77:0/3110503998 >> 172.21.15.77:0/3110503998 conn(0x7fb87806b720 msgr2=0x7fb8780ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:29.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.328+0000 7fb87eea5640 1 -- 172.21.15.77:0/3110503998 shutdown_connections 2024-04-25T01:22:29.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.328+0000 7fb87eea5640 1 -- 172.21.15.77:0/3110503998 wait complete. 2024-04-25T01:22:29.320 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:29.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.438+0000 7f7b273fb640 1 Processor -- start 2024-04-25T01:22:29.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.438+0000 7f7b273fb640 1 -- start start 2024-04-25T01:22:29.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.438+0000 7f7b273fb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b200d23d0 0x7f7b200d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b273fb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200b9a90 0x7f7b200d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b273fb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b200cea40 con 0x7f7b200d27b0 2024-04-25T01:22:29.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b273fb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f7b200cece0 con 0x7f7b200d23d0 2024-04-25T01:22:29.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b273fb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7b200cefc0 con 0x7f7b200b9a90 2024-04-25T01:22:29.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b25197640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b200d23d0 0x7f7b200d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b24996640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7b200d27b0 0x7f7b200b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57416/0 (socket says 172.21.15.77:57416) 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b24996640 1 -- 172.21.15.77:0/257460793 learned_addr learned my addr 172.21.15.77:0/257460793 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b25998640 1 --2- 172.21.15.77:0/257460793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200b9a90 0x7f7b200d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b25197640 1 -- 172.21.15.77:0/257460793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200b9a90 msgr2=0x7f7b200d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b25197640 1 --2- 172.21.15.77:0/257460793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200b9a90 0x7f7b200d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b25197640 1 -- 172.21.15.77:0/257460793 >> v1:172.21.15.77:6789/0 conn(0x7f7b200d27b0 legacy=0x7f7b200b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b25197640 1 -- 172.21.15.77:0/257460793 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b200cf2a0 con 0x7f7b200d23d0 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.439+0000 7f7b25998640 1 --2- 172.21.15.77:0/257460793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200b9a90 0x7f7b200d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b25197640 1 --2- 172.21.15.77:0/257460793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b200d23d0 0x7f7b200d4920 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0097a0 tx=0x7f7b1c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36102c846775668e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b17fff640 1 -- 172.21.15.77:0/257460793 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c037070 con 0x7f7b200d23d0 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b17fff640 1 -- 172.21.15.77:0/257460793 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b1c032440 con 0x7f7b200d23d0 2024-04-25T01:22:29.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b17fff640 1 -- 172.21.15.77:0/257460793 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c032810 con 0x7f7b200d23d0 2024-04-25T01:22:29.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b273fb640 1 -- 172.21.15.77:0/257460793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b200d23d0 msgr2=0x7f7b200d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b273fb640 1 --2- 172.21.15.77:0/257460793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b200d23d0 0x7f7b200d4920 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0097a0 tx=0x7f7b1c02a210 comp rx=0 tx=0).stop 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b273fb640 1 -- 172.21.15.77:0/257460793 shutdown_connections 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b273fb640 1 --2- 172.21.15.77:0/257460793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200b9a90 0x7f7b200d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b273fb640 1 --2- 172.21.15.77:0/257460793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b200d23d0 0x7f7b200d4920 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b273fb640 1 -- 172.21.15.77:0/257460793 >> 172.21.15.77:0/257460793 conn(0x7f7b200d3e30 msgr2=0x7f7b200d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b273fb640 1 -- 172.21.15.77:0/257460793 shutdown_connections 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.440+0000 7f7b273fb640 1 -- 172.21.15.77:0/257460793 wait complete. 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.441+0000 7f7b273fb640 1 Processor -- start 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b273fb640 1 -- start start 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b273fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b200b9a90 0x7f7b200ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b273fb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200d23d0 0x7f7b20151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b273fb640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b200d27b0 0x7f7b200caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b273fb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b200cfd80 con 0x7f7b200b9a90 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b273fb640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7b200cf860 con 0x7f7b200d27b0 2024-04-25T01:22:29.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b273fb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7b200b6c80 con 0x7f7b200d23d0 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b25197640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b200b9a90 0x7f7b200ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b25197640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b200b9a90 0x7f7b200ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48988/0 (socket says 172.21.15.77:48988) 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b25197640 1 -- 172.21.15.77:0/3789707714 learned_addr learned my addr 172.21.15.77:0/3789707714 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b24996640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200d23d0 0x7f7b20151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.442+0000 7f7b25998640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b200d27b0 0x7f7b200caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b25197640 1 -- 172.21.15.77:0/3789707714 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200d23d0 msgr2=0x7f7b20151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b25197640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200d23d0 0x7f7b20151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b25197640 1 -- 172.21.15.77:0/3789707714 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b200d27b0 msgr2=0x7f7b200caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b25197640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b200d27b0 0x7f7b200caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b25197640 1 -- 172.21.15.77:0/3789707714 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b200cc230 con 0x7f7b200b9a90 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b24996640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200d23d0 0x7f7b20151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b25998640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b200d27b0 0x7f7b200caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b25197640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b200b9a90 0x7f7b200ca120 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c035040 tx=0x7f7b1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b15ffb640 1 -- 172.21.15.77:0/3789707714 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c037040 con 0x7f7b200b9a90 2024-04-25T01:22:29.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b15ffb640 1 -- 172.21.15.77:0/3789707714 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b1c030770 con 0x7f7b200b9a90 2024-04-25T01:22:29.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b273fb640 1 -- 172.21.15.77:0/3789707714 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b200ca700 con 0x7f7b200b9a90 2024-04-25T01:22:29.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b15ffb640 1 -- 172.21.15.77:0/3789707714 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c030b40 con 0x7f7b200b9a90 2024-04-25T01:22:29.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.443+0000 7f7b273fb640 1 -- 172.21.15.77:0/3789707714 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b200cd3c0 con 0x7f7b200b9a90 2024-04-25T01:22:29.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.444+0000 7f7b273fb640 1 -- 172.21.15.77:0/3789707714 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b20158fc0 con 0x7f7b200b9a90 2024-04-25T01:22:29.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.444+0000 7f7b15ffb640 1 -- 172.21.15.77:0/3789707714 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7b1c044020 con 0x7f7b200b9a90 2024-04-25T01:22:29.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.445+0000 7f7b15ffb640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7b0406be30 0x7f7b0406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.445+0000 7f7b15ffb640 1 -- 172.21.15.77:0/3789707714 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f7b1c0ab7f0 con 0x7f7b200b9a90 2024-04-25T01:22:29.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.445+0000 7f7b24996640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7b0406be30 0x7f7b0406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.445+0000 7f7b24996640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7b0406be30 0x7f7b0406e2f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7b200d4890 tx=0x7f7b10009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:29.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.449+0000 7f7b15ffb640 1 -- 172.21.15.77:0/3789707714 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7b20158fc0 con 0x7f7b200b9a90 2024-04-25T01:22:29.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.621+0000 7f7b273fb640 1 -- 172.21.15.77:0/3789707714 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7b20068fb0 con 0x7f7b200b9a90 2024-04-25T01:22:29.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.621+0000 7f7b15ffb640 1 -- 172.21.15.77:0/3789707714 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f7b1c032e20 con 0x7f7b200b9a90 2024-04-25T01:22:29.604 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.623+0000 7f7b273fb640 1 -- 172.21.15.77:0/3789707714 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7b0406be30 msgr2=0x7f7b0406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.623+0000 7f7b273fb640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7b0406be30 0x7f7b0406e2f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7b200d4890 tx=0x7f7b10009040 comp rx=0 tx=0).stop 2024-04-25T01:22:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.623+0000 7f7b273fb640 1 -- 172.21.15.77:0/3789707714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b200b9a90 msgr2=0x7f7b200ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.623+0000 7f7b273fb640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b200b9a90 0x7f7b200ca120 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c035040 tx=0x7f7b1c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.623+0000 7f7b273fb640 1 -- 172.21.15.77:0/3789707714 shutdown_connections 2024-04-25T01:22:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.623+0000 7f7b273fb640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7b0406be30 0x7f7b0406e2f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.623+0000 7f7b273fb640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b200d27b0 0x7f7b200caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.623+0000 7f7b273fb640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b200d23d0 0x7f7b20151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.623+0000 7f7b273fb640 1 --2- 172.21.15.77:0/3789707714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b200b9a90 0x7f7b200ca120 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.623+0000 7f7b273fb640 1 -- 172.21.15.77:0/3789707714 >> 172.21.15.77:0/3789707714 conn(0x7f7b200d3e30 msgr2=0x7f7b200b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.624+0000 7f7b273fb640 1 -- 172.21.15.77:0/3789707714 shutdown_connections 2024-04-25T01:22:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.624+0000 7f7b273fb640 1 -- 172.21.15.77:0/3789707714 wait complete. 2024-04-25T01:22:29.616 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:29.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.735+0000 7f518b86b640 1 Processor -- start 2024-04-25T01:22:29.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.735+0000 7f518b86b640 1 -- start start 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f518b86b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5184158a70 0x7f51840b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f518b86b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f51840b37c0 0x7f51840b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f518b86b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f518415c210 con 0x7f518415b520 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f518b86b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f51840b2ce0 con 0x7f51840b37c0 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f518b86b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f51840b2fc0 con 0x7f5184158a70 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f5188e06640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f51840b37c0 0x7f51840b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f5188e06640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f51840b37c0 0x7f51840b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48996/0 (socket says 172.21.15.77:48996) 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f5188e06640 1 -- 172.21.15.77:0/279717994 learned_addr learned my addr 172.21.15.77:0/279717994 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f5189607640 1 --2- 172.21.15.77:0/279717994 >> v2:172.21.15.121:3300/0 conn(0x7f5184158a70 0x7f51840b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f5188e06640 1 -- 172.21.15.77:0/279717994 >> v2:172.21.15.121:3300/0 conn(0x7f5184158a70 msgr2=0x7f51840b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f5188e06640 1 --2- 172.21.15.77:0/279717994 >> v2:172.21.15.121:3300/0 conn(0x7f5184158a70 0x7f51840b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f5188e06640 1 -- 172.21.15.77:0/279717994 >> v1:172.21.15.121:6789/0 conn(0x7f518415b520 legacy=0x7f518415b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.736+0000 7f5188e06640 1 -- 172.21.15.77:0/279717994 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51840b32a0 con 0x7f51840b37c0 2024-04-25T01:22:29.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.737+0000 7f5188e06640 1 --2- 172.21.15.77:0/279717994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51840b37c0 0x7f51840b4600 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5174009870 tx=0x7f517402a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba7faa3f3d1c405 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:29.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.737+0000 7f517bfff640 1 -- 172.21.15.77:0/279717994 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174037070 con 0x7f51840b37c0 2024-04-25T01:22:29.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.737+0000 7f517bfff640 1 -- 172.21.15.77:0/279717994 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5174032440 con 0x7f51840b37c0 2024-04-25T01:22:29.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.737+0000 7f517bfff640 1 -- 172.21.15.77:0/279717994 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174032810 con 0x7f51840b37c0 2024-04-25T01:22:29.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.737+0000 7f518b86b640 1 -- 172.21.15.77:0/279717994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51840b37c0 msgr2=0x7f51840b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.737+0000 7f518b86b640 1 --2- 172.21.15.77:0/279717994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51840b37c0 0x7f51840b4600 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5174009870 tx=0x7f517402a2a0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.737+0000 7f518b86b640 1 -- 172.21.15.77:0/279717994 shutdown_connections 2024-04-25T01:22:29.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.737+0000 7f518b86b640 1 --2- 172.21.15.77:0/279717994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51840b37c0 0x7f51840b4600 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.737+0000 7f518b86b640 1 --2- 172.21.15.77:0/279717994 >> v2:172.21.15.121:3300/0 conn(0x7f5184158a70 0x7f51840b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.737+0000 7f518b86b640 1 -- 172.21.15.77:0/279717994 >> 172.21.15.77:0/279717994 conn(0x7f518406b720 msgr2=0x7f51840b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:29.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.738+0000 7f518b86b640 1 -- 172.21.15.77:0/279717994 shutdown_connections 2024-04-25T01:22:29.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.738+0000 7f518b86b640 1 -- 172.21.15.77:0/279717994 wait complete. 2024-04-25T01:22:29.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.739+0000 7f518b86b640 1 Processor -- start 2024-04-25T01:22:29.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.739+0000 7f518b86b640 1 -- start start 2024-04-25T01:22:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.739+0000 7f518b86b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51840b37c0 0x7f51840b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.739+0000 7f518b86b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5184158a70 0x7f518413c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.739+0000 7f518b86b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f518415b520 0x7f51840ce340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.739+0000 7f518b86b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f518406cb70 con 0x7f5184158a70 2024-04-25T01:22:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.739+0000 7f518b86b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f518406c9f0 con 0x7f518415b520 2024-04-25T01:22:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.739+0000 7f518b86b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f518406ccf0 con 0x7f51840b37c0 2024-04-25T01:22:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.739+0000 7f5188e06640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5184158a70 0x7f518413c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.739+0000 7f5188e06640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5184158a70 0x7f518413c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49006/0 (socket says 172.21.15.77:49006) 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.739+0000 7f5188e06640 1 -- 172.21.15.77:0/2533323048 learned_addr learned my addr 172.21.15.77:0/2533323048 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5189607640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51840b37c0 0x7f51840b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5189e08640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f518415b520 0x7f51840ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5188e06640 1 -- 172.21.15.77:0/2533323048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51840b37c0 msgr2=0x7f51840b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5188e06640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51840b37c0 0x7f51840b02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5188e06640 1 -- 172.21.15.77:0/2533323048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f518415b520 msgr2=0x7f51840ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5188e06640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f518415b520 0x7f51840ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5188e06640 1 -- 172.21.15.77:0/2533323048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51840af850 con 0x7f5184158a70 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5189607640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51840b37c0 0x7f51840b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5188e06640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5184158a70 0x7f518413c460 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5174000c00 tx=0x7f51740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5179ffb640 1 -- 172.21.15.77:0/2533323048 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174037040 con 0x7f5184158a70 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5179ffb640 1 -- 172.21.15.77:0/2533323048 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5174042070 con 0x7f5184158a70 2024-04-25T01:22:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f518b86b640 1 -- 172.21.15.77:0/2533323048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f518413cf40 con 0x7f5184158a70 2024-04-25T01:22:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f5179ffb640 1 -- 172.21.15.77:0/2533323048 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174030880 con 0x7f5184158a70 2024-04-25T01:22:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.740+0000 7f518b86b640 1 -- 172.21.15.77:0/2533323048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51840afda0 con 0x7f5184158a70 2024-04-25T01:22:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.741+0000 7f518b86b640 1 -- 172.21.15.77:0/2533323048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5184068fb0 con 0x7f5184158a70 2024-04-25T01:22:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.742+0000 7f5179ffb640 1 -- 172.21.15.77:0/2533323048 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5174030a20 con 0x7f5184158a70 2024-04-25T01:22:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.742+0000 7f5179ffb640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f516806be60 0x7f516806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.742+0000 7f5179ffb640 1 -- 172.21.15.77:0/2533323048 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f51740abab0 con 0x7f5184158a70 2024-04-25T01:22:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.742+0000 7f5189607640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f516806be60 0x7f516806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.743+0000 7f5189607640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f516806be60 0x7f516806e320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f517c009b90 tx=0x7f517c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:29.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.746+0000 7f5179ffb640 1 -- 172.21.15.77:0/2533323048 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f517407a050 con 0x7f5184158a70 2024-04-25T01:22:29.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.920+0000 7f518b86b640 1 -- 172.21.15.77:0/2533323048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5184157ec0 con 0x7f5184158a70 2024-04-25T01:22:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.920+0000 7f5179ffb640 1 -- 172.21.15.77:0/2533323048 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f5174079950 con 0x7f5184158a70 2024-04-25T01:22:29.902 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:29.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 -- 172.21.15.77:0/2533323048 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f516806be60 msgr2=0x7f516806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f516806be60 0x7f516806e320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f517c009b90 tx=0x7f517c008040 comp rx=0 tx=0).stop 2024-04-25T01:22:29.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 -- 172.21.15.77:0/2533323048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5184158a70 msgr2=0x7f518413c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5184158a70 0x7f518413c460 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5174000c00 tx=0x7f51740046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 -- 172.21.15.77:0/2533323048 shutdown_connections 2024-04-25T01:22:29.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f516806be60 0x7f516806e320 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5184158a70 0x7f518413c460 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f518415b520 0x7f51840ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 --2- 172.21.15.77:0/2533323048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51840b37c0 0x7f51840b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 -- 172.21.15.77:0/2533323048 >> 172.21.15.77:0/2533323048 conn(0x7f518406b720 msgr2=0x7f518415a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:29.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 -- 172.21.15.77:0/2533323048 shutdown_connections 2024-04-25T01:22:29.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:29.922+0000 7f518b86b640 1 -- 172.21.15.77:0/2533323048 wait complete. 2024-04-25T01:22:29.915 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:30.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.032+0000 7f862fd42640 1 Processor -- start 2024-04-25T01:22:30.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.033+0000 7f862fd42640 1 -- start start 2024-04-25T01:22:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.033+0000 7f862fd42640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86280d23d0 0x7f86280d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.033+0000 7f862fd42640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f86280b9a90 0x7f86280d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.033+0000 7f862fd42640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86280cea40 con 0x7f86280d27b0 2024-04-25T01:22:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.033+0000 7f862fd42640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f86280cece0 con 0x7f86280d23d0 2024-04-25T01:22:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.033+0000 7f862fd42640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f86280cefc0 con 0x7f86280b9a90 2024-04-25T01:22:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.034+0000 7f862dade640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86280d23d0 0x7f86280d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.034+0000 7f862d2dd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f86280d27b0 0x7f86280b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57422/0 (socket says 172.21.15.77:57422) 2024-04-25T01:22:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.034+0000 7f862d2dd640 1 -- 172.21.15.77:0/2313228850 learned_addr learned my addr 172.21.15.77:0/2313228850 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.034+0000 7f862e2df640 1 --2- 172.21.15.77:0/2313228850 >> v2:172.21.15.121:3300/0 conn(0x7f86280b9a90 0x7f86280d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.034+0000 7f862dade640 1 -- 172.21.15.77:0/2313228850 >> v2:172.21.15.121:3300/0 conn(0x7f86280b9a90 msgr2=0x7f86280d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.034+0000 7f862dade640 1 --2- 172.21.15.77:0/2313228850 >> v2:172.21.15.121:3300/0 conn(0x7f86280b9a90 0x7f86280d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.034+0000 7f862dade640 1 -- 172.21.15.77:0/2313228850 >> v1:172.21.15.77:6789/0 conn(0x7f86280d27b0 legacy=0x7f86280b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.034+0000 7f862cadc640 1 -- 172.21.15.77:0/2313228850 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2603869105 0 0) 0x7f86280cea40 con 0x7f86280d27b0 2024-04-25T01:22:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.034+0000 7f862dade640 1 -- 172.21.15.77:0/2313228850 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86280cf2a0 con 0x7f86280d23d0 2024-04-25T01:22:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.034+0000 7f862dade640 1 --2- 172.21.15.77:0/2313228850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86280d23d0 0x7f86280d4920 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f862806b140 tx=0x7f862402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f4810d9eb8daeb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.035+0000 7f862cadc640 1 -- 172.21.15.77:0/2313228850 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8624037070 con 0x7f86280d23d0 2024-04-25T01:22:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.035+0000 7f862cadc640 1 -- 172.21.15.77:0/2313228850 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8624032440 con 0x7f86280d23d0 2024-04-25T01:22:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.035+0000 7f862cadc640 1 -- 172.21.15.77:0/2313228850 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8624032810 con 0x7f86280d23d0 2024-04-25T01:22:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.035+0000 7f862fd42640 1 -- 172.21.15.77:0/2313228850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86280d23d0 msgr2=0x7f86280d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.035+0000 7f862fd42640 1 --2- 172.21.15.77:0/2313228850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86280d23d0 0x7f86280d4920 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f862806b140 tx=0x7f862402a230 comp rx=0 tx=0).stop 2024-04-25T01:22:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.035+0000 7f862fd42640 1 -- 172.21.15.77:0/2313228850 shutdown_connections 2024-04-25T01:22:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.035+0000 7f862fd42640 1 --2- 172.21.15.77:0/2313228850 >> v2:172.21.15.121:3300/0 conn(0x7f86280b9a90 0x7f86280d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.035+0000 7f862fd42640 1 --2- 172.21.15.77:0/2313228850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86280d23d0 0x7f86280d4920 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.035+0000 7f862fd42640 1 -- 172.21.15.77:0/2313228850 >> 172.21.15.77:0/2313228850 conn(0x7f86280d3e30 msgr2=0x7f86280d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.035+0000 7f862fd42640 1 -- 172.21.15.77:0/2313228850 shutdown_connections 2024-04-25T01:22:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.035+0000 7f862fd42640 1 -- 172.21.15.77:0/2313228850 wait complete. 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.036+0000 7f862fd42640 1 Processor -- start 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.036+0000 7f862fd42640 1 -- start start 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862fd42640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86280b9a90 0x7f8628159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862fd42640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86280d23d0 0x7f86281e6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862fd42640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280d27b0 0x7f86281f7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862fd42640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86280cfd80 con 0x7f86280d23d0 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862fd42640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f86280cf860 con 0x7f86280d27b0 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862fd42640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f86280b6c80 con 0x7f86280b9a90 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862d2dd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86280d23d0 0x7f86281e6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862d2dd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86280d23d0 0x7f86281e6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49028/0 (socket says 172.21.15.77:49028) 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862d2dd640 1 -- 172.21.15.77:0/4158863389 learned_addr learned my addr 172.21.15.77:0/4158863389 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862dade640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86280b9a90 0x7f8628159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862e2df640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280d27b0 0x7f86281f7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862d2dd640 1 -- 172.21.15.77:0/4158863389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86280b9a90 msgr2=0x7f8628159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.037+0000 7f862d2dd640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86280b9a90 0x7f8628159cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.038+0000 7f862d2dd640 1 -- 172.21.15.77:0/4158863389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280d27b0 msgr2=0x7f86281f7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.038+0000 7f862d2dd640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280d27b0 0x7f86281f7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.038+0000 7f862d2dd640 1 -- 172.21.15.77:0/4158863389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86280c8ae0 con 0x7f86280d23d0 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.038+0000 7f862dade640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86280b9a90 0x7f8628159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.038+0000 7f862e2df640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280d27b0 0x7f86281f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.038+0000 7f862d2dd640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86280d23d0 0x7f86281e6ff0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f861800c430 tx=0x7f861800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.038+0000 7f861e7fc640 1 -- 172.21.15.77:0/4158863389 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618017070 con 0x7f86280d23d0 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.038+0000 7f861e7fc640 1 -- 172.21.15.77:0/4158863389 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8618005ce0 con 0x7f86280d23d0 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.038+0000 7f861e7fc640 1 -- 172.21.15.77:0/4158863389 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618007310 con 0x7f86280d23d0 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.038+0000 7f862fd42640 1 -- 172.21.15.77:0/4158863389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86280c8dc0 con 0x7f86280d23d0 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.038+0000 7f862fd42640 1 -- 172.21.15.77:0/4158863389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86281ea460 con 0x7f86280d23d0 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.039+0000 7f862fd42640 1 -- 172.21.15.77:0/4158863389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8628067390 con 0x7f86280d23d0 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.039+0000 7f861e7fc640 1 -- 172.21.15.77:0/4158863389 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f861801a070 con 0x7f86280d23d0 2024-04-25T01:22:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.040+0000 7f861e7fc640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860c06be30 0x7f860c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.040+0000 7f861e7fc640 1 -- 172.21.15.77:0/4158863389 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f861808a620 con 0x7f86280d23d0 2024-04-25T01:22:30.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.040+0000 7f862dade640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860c06be30 0x7f860c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.040+0000 7f862dade640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860c06be30 0x7f860c06e2f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8624034000 tx=0x7f862402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.044+0000 7f861e7fc640 1 -- 172.21.15.77:0/4158863389 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8628067390 con 0x7f86280d23d0 2024-04-25T01:22:30.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.221+0000 7f862fd42640 1 -- 172.21.15.77:0/4158863389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8628068fb0 con 0x7f86280d23d0 2024-04-25T01:22:30.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.221+0000 7f861e7fc640 1 -- 172.21.15.77:0/4158863389 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f8618058c50 con 0x7f86280d23d0 2024-04-25T01:22:30.206 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:30.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.223+0000 7f862fd42640 1 -- 172.21.15.77:0/4158863389 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860c06be30 msgr2=0x7f860c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.223+0000 7f862fd42640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860c06be30 0x7f860c06e2f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8624034000 tx=0x7f862402e000 comp rx=0 tx=0).stop 2024-04-25T01:22:30.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.223+0000 7f862fd42640 1 -- 172.21.15.77:0/4158863389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86280d23d0 msgr2=0x7f86281e6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.223+0000 7f862fd42640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86280d23d0 0x7f86281e6ff0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f861800c430 tx=0x7f861800c8f0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.223+0000 7f862fd42640 1 -- 172.21.15.77:0/4158863389 shutdown_connections 2024-04-25T01:22:30.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.223+0000 7f862fd42640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f860c06be30 0x7f860c06e2f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.223+0000 7f862fd42640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86280d23d0 0x7f86281e6ff0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.223+0000 7f862fd42640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86280d27b0 0x7f86281f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.223+0000 7f862fd42640 1 --2- 172.21.15.77:0/4158863389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86280b9a90 0x7f8628159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.223+0000 7f862fd42640 1 -- 172.21.15.77:0/4158863389 >> 172.21.15.77:0/4158863389 conn(0x7f86280d3e30 msgr2=0x7f86280b6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:30.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.224+0000 7f862fd42640 1 -- 172.21.15.77:0/4158863389 shutdown_connections 2024-04-25T01:22:30.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.224+0000 7f862fd42640 1 -- 172.21.15.77:0/4158863389 wait complete. 2024-04-25T01:22:30.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:30.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.333+0000 7f06d0aa5640 1 Processor -- start 2024-04-25T01:22:30.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.333+0000 7f06d0aa5640 1 -- start start 2024-04-25T01:22:30.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06d0aa5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06cc15add0 0x7f06cc15f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06d0aa5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06cc0af960 0x7f06cc06c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06d0aa5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f06cc06c670 0x7f06cc0b4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06d0aa5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f06cc06cac0 con 0x7f06cc15add0 2024-04-25T01:22:30.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06d0aa5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f06cc15b1b0 con 0x7f06cc06c670 2024-04-25T01:22:30.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06d0aa5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f06cc15b440 con 0x7f06cc0af960 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06ca59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06cc15add0 0x7f06cc15f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06ca59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06cc15add0 0x7f06cc15f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49034/0 (socket says 172.21.15.77:49034) 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06ca59c640 1 -- 172.21.15.77:0/1269038864 learned_addr learned my addr 172.21.15.77:0/1269038864 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06ca59c640 1 -- 172.21.15.77:0/1269038864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06cc0af960 msgr2=0x7f06cc06c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06ca59c640 1 --2- 172.21.15.77:0/1269038864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06cc0af960 0x7f06cc06c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06ca59c640 1 -- 172.21.15.77:0/1269038864 >> v2:172.21.15.121:3300/0 conn(0x7f06cc06c670 msgr2=0x7f06cc0b4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06ca59c640 1 --2- 172.21.15.77:0/1269038864 >> v2:172.21.15.121:3300/0 conn(0x7f06cc06c670 0x7f06cc0b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.334+0000 7f06ca59c640 1 -- 172.21.15.77:0/1269038864 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06cc15b700 con 0x7f06cc15add0 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.335+0000 7f06ca59c640 1 --2- 172.21.15.77:0/1269038864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06cc15add0 0x7f06cc15f250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0685f0 tx=0x7f06bc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871e53fe13c19790 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.335+0000 7f06c9d9b640 1 -- 172.21.15.77:0/1269038864 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc037070 con 0x7f06cc15add0 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.335+0000 7f06c9d9b640 1 -- 172.21.15.77:0/1269038864 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06bc032440 con 0x7f06cc15add0 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.335+0000 7f06c9d9b640 1 -- 172.21.15.77:0/1269038864 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc032810 con 0x7f06cc15add0 2024-04-25T01:22:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.335+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1269038864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06cc15add0 msgr2=0x7f06cc15f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.335+0000 7f06d0aa5640 1 --2- 172.21.15.77:0/1269038864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06cc15add0 0x7f06cc15f250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0685f0 tx=0x7f06bc02a1e0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.336+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1269038864 shutdown_connections 2024-04-25T01:22:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.336+0000 7f06d0aa5640 1 --2- 172.21.15.77:0/1269038864 >> v2:172.21.15.121:3300/0 conn(0x7f06cc06c670 0x7f06cc0b4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.336+0000 7f06d0aa5640 1 --2- 172.21.15.77:0/1269038864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06cc0af960 0x7f06cc06c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.336+0000 7f06d0aa5640 1 --2- 172.21.15.77:0/1269038864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06cc15add0 0x7f06cc15f250 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.336+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1269038864 >> 172.21.15.77:0/1269038864 conn(0x7f06cc071550 msgr2=0x7f06cc071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.336+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1269038864 shutdown_connections 2024-04-25T01:22:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.336+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1269038864 wait complete. 2024-04-25T01:22:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.337+0000 7f06d0aa5640 1 Processor -- start 2024-04-25T01:22:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.337+0000 7f06d0aa5640 1 -- start start 2024-04-25T01:22:30.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06d0aa5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc06c670 0x7f06cc146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06d0aa5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06cc0af960 0x7f06cc151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06d0aa5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06cc15add0 0x7f06cc147610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06d0aa5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc15bfa0 con 0x7f06cc15add0 2024-04-25T01:22:30.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06d0aa5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc15bcc0 con 0x7f06cc06c670 2024-04-25T01:22:30.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06d0aa5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f06cc0b4b80 con 0x7f06cc0af960 2024-04-25T01:22:30.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06cad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06cc15add0 0x7f06cc147610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06cad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06cc15add0 0x7f06cc147610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49042/0 (socket says 172.21.15.77:49042) 2024-04-25T01:22:30.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06cad9d640 1 -- 172.21.15.77:0/1452607454 learned_addr learned my addr 172.21.15.77:0/1452607454 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:30.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06ca59c640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc06c670 0x7f06cc146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06c3fff640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06cc0af960 0x7f06cc151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06cad9d640 1 -- 172.21.15.77:0/1452607454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06cc0af960 msgr2=0x7f06cc151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06cad9d640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06cc0af960 0x7f06cc151640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06cad9d640 1 -- 172.21.15.77:0/1452607454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc06c670 msgr2=0x7f06cc146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06cad9d640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc06c670 0x7f06cc146cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06cad9d640 1 -- 172.21.15.77:0/1452607454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06cc0ada60 con 0x7f06cc15add0 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06ca59c640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc06c670 0x7f06cc146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06cad9d640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06cc15add0 0x7f06cc147610 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f06c400b970 tx=0x7f06c400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06c2ffd640 1 -- 172.21.15.77:0/1452607454 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4019070 con 0x7f06cc15add0 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06c2ffd640 1 -- 172.21.15.77:0/1452607454 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c400cb60 con 0x7f06cc15add0 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06c2ffd640 1 -- 172.21.15.77:0/1452607454 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4014440 con 0x7f06cc15add0 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1452607454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06cc0add40 con 0x7f06cc15add0 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.338+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1452607454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06cc147200 con 0x7f06cc15add0 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.339+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1452607454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06cc068890 con 0x7f06cc15add0 2024-04-25T01:22:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.340+0000 7f06c2ffd640 1 -- 172.21.15.77:0/1452607454 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f06c401c070 con 0x7f06cc15add0 2024-04-25T01:22:30.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.340+0000 7f06c2ffd640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f06ac06be30 0x7f06ac06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.340+0000 7f06c2ffd640 1 -- 172.21.15.77:0/1452607454 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f06c408afe0 con 0x7f06cc15add0 2024-04-25T01:22:30.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.340+0000 7f06ca59c640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f06ac06be30 0x7f06ac06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.341+0000 7f06ca59c640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f06ac06be30 0x7f06ac06e2f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f06bc034000 tx=0x7f06bc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.344+0000 7f06c2ffd640 1 -- 172.21.15.77:0/1452607454 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f06cc068890 con 0x7f06cc15add0 2024-04-25T01:22:30.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.521+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1452607454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06cc072320 con 0x7f06cc15add0 2024-04-25T01:22:30.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.522+0000 7f06c2ffd640 1 -- 172.21.15.77:0/1452607454 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18922 (secure 0 0 0) 0x7f06c4059580 con 0x7f06cc15add0 2024-04-25T01:22:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:30.504 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:26.287128+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1452607454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f06ac06be30 msgr2=0x7f06ac06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f06ac06be30 0x7f06ac06e2f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f06bc034000 tx=0x7f06bc02e000 comp rx=0 tx=0).stop 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1452607454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06cc15add0 msgr2=0x7f06cc147610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06cc15add0 0x7f06cc147610 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f06c400b970 tx=0x7f06c400be30 comp rx=0 tx=0).stop 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1452607454 shutdown_connections 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f06ac06be30 0x7f06ac06e2f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06cc15add0 0x7f06cc147610 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06cc0af960 0x7f06cc151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 --2- 172.21.15.77:0/1452607454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc06c670 0x7f06cc146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1452607454 >> 172.21.15.77:0/1452607454 conn(0x7f06cc071550 msgr2=0x7f06cc150af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1452607454 shutdown_connections 2024-04-25T01:22:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.524+0000 7f06d0aa5640 1 -- 172.21.15.77:0/1452607454 wait complete. 2024-04-25T01:22:30.516 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:30.517 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:30.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.635+0000 7f47e7ff9640 1 Processor -- start 2024-04-25T01:22:30.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.636+0000 7f47e7ff9640 1 -- start start 2024-04-25T01:22:30.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.636+0000 7f47e7ff9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e00af8c0 0x7f47e00cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.636+0000 7f47e7ff9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f47e00b5060 0x7f47e006b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.636+0000 7f47e7ff9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e00b0200 con 0x7f47e006b660 2024-04-25T01:22:30.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.636+0000 7f47e7ff9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f47e006ba90 con 0x7f47e00b5060 2024-04-25T01:22:30.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.636+0000 7f47e7ff9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f47e00b5840 con 0x7f47e00af8c0 2024-04-25T01:22:30.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e6596640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f47e006b660 0x7f47e015d940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57430/0 (socket says 172.21.15.77:57430) 2024-04-25T01:22:30.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e6596640 1 -- 172.21.15.77:0/1061322062 learned_addr learned my addr 172.21.15.77:0/1061322062 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:30.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 594274670 0 0) 0x7f47e00b0200 con 0x7f47e006b660 2024-04-25T01:22:30.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e5594640 1 --2- 172.21.15.77:0/1061322062 >> v2:172.21.15.121:3300/0 conn(0x7f47e00b5060 0x7f47e006b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e5d95640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e00af8c0 0x7f47e00cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47cc003520 con 0x7f47e006b660 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 640429289 0 0) 0x7f47cc003520 con 0x7f47e006b660 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e00af8c0 msgr2=0x7f47e00cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e00af8c0 0x7f47e00cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 >> v2:172.21.15.121:3300/0 conn(0x7f47e00b5060 msgr2=0x7f47e006b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 --2- 172.21.15.77:0/1061322062 >> v2:172.21.15.121:3300/0 conn(0x7f47e00b5060 0x7f47e006b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e00b5b20 con 0x7f47e006b660 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f47dc001690 con 0x7f47e006b660 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 >> v1:172.21.15.77:6789/0 conn(0x7f47e006b660 legacy=0x7f47e015d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47cc0039b0 0x7f47cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e00af8c0 0x7f47cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47cc004d40 0x7f47cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f47e006ba90 con 0x7f47cc0039b0 2024-04-25T01:22:30.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f47e00b5840 con 0x7f47cc004d40 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f47cc00b9c0 con 0x7f47e00af8c0 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e5d95640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47cc0039b0 0x7f47cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e5594640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e00af8c0 0x7f47cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e6596640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47cc004d40 0x7f47cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e5d95640 1 -- 172.21.15.77:0/1061322062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e00af8c0 msgr2=0x7f47cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e5d95640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e00af8c0 0x7f47cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e5d95640 1 -- 172.21.15.77:0/1061322062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47cc004d40 msgr2=0x7f47cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.637+0000 7f47e5d95640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47cc004d40 0x7f47cc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.638+0000 7f47e5d95640 1 -- 172.21.15.77:0/1061322062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47cc003520 con 0x7f47cc0039b0 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.638+0000 7f47e5d95640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47cc0039b0 0x7f47cc003dd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f47d8009fc0 tx=0x7f47cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3edc765cfc4c91a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.638+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d8009c10 con 0x7f47cc0039b0 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.638+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d8036070 con 0x7f47cc0039b0 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.638+0000 7f47e4d93640 1 -- 172.21.15.77:0/1061322062 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d8031880 con 0x7f47cc0039b0 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.639+0000 7f47e7ff9640 1 -- 172.21.15.77:0/1061322062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47cc0039b0 msgr2=0x7f47cc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.639+0000 7f47e7ff9640 1 --2- 172.21.15.77:0/1061322062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47cc0039b0 0x7f47cc003dd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f47d8009fc0 tx=0x7f47cc005830 comp rx=0 tx=0).stop 2024-04-25T01:22:30.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.639+0000 7f47e6596640 1 -- 172.21.15.77:0/1061322062 reap_dead start 2024-04-25T01:22:30.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.639+0000 7f47e7ff9640 1 -- 172.21.15.77:0/1061322062 shutdown_connections 2024-04-25T01:22:30.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.639+0000 7f47e7ff9640 1 -- 172.21.15.77:0/1061322062 >> 172.21.15.77:0/1061322062 conn(0x7f47e00b1f00 msgr2=0x7f47e00b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:30.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.639+0000 7f47e7ff9640 1 -- 172.21.15.77:0/1061322062 shutdown_connections 2024-04-25T01:22:30.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.639+0000 7f47e7ff9640 1 -- 172.21.15.77:0/1061322062 wait complete. 2024-04-25T01:22:30.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.640+0000 7f47e7ff9640 1 Processor -- start 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.640+0000 7f47e7ff9640 1 -- start start 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e7ff9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47e015b490 0x7f47e00c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e7ff9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47e00c8700 0x7f47e015d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e7ff9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e015d630 0x7f47e006e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e7ff9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f47e00b5510 con 0x7f47e00c8700 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e7ff9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f47e00b60e0 con 0x7f47e015b490 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e7ff9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f47e00afca0 con 0x7f47e015d630 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e5594640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47e00c8700 0x7f47e015d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e5594640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47e00c8700 0x7f47e015d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49060/0 (socket says 172.21.15.77:49060) 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e5594640 1 -- 172.21.15.77:0/164443780 learned_addr learned my addr 172.21.15.77:0/164443780 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e5d95640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47e015b490 0x7f47e00c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e6596640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e015d630 0x7f47e006e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e5594640 1 -- 172.21.15.77:0/164443780 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e015d630 msgr2=0x7f47e006e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e5594640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e015d630 0x7f47e006e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e5594640 1 -- 172.21.15.77:0/164443780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47e015b490 msgr2=0x7f47e00c81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e5594640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47e015b490 0x7f47e00c81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e5594640 1 -- 172.21.15.77:0/164443780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e00ae0f0 con 0x7f47e00c8700 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e5d95640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47e015b490 0x7f47e00c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e6596640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e015d630 0x7f47e006e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.641+0000 7f47e5594640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47e00c8700 0x7f47e015d0f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f47d0011700 tx=0x7f47d0011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.642+0000 7f47d67fc640 1 -- 172.21.15.77:0/164443780 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d000f9c0 con 0x7f47e00c8700 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.642+0000 7f47d67fc640 1 -- 172.21.15.77:0/164443780 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d000fb60 con 0x7f47e00c8700 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.642+0000 7f47e7ff9640 1 -- 172.21.15.77:0/164443780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e00c7400 con 0x7f47e00c8700 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.642+0000 7f47d67fc640 1 -- 172.21.15.77:0/164443780 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d0015460 con 0x7f47e00c8700 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.642+0000 7f47e7ff9640 1 -- 172.21.15.77:0/164443780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e00c79c0 con 0x7f47e00c8700 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.643+0000 7f47e7ff9640 1 -- 172.21.15.77:0/164443780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47e0068fb0 con 0x7f47e00c8700 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.643+0000 7f47d67fc640 1 -- 172.21.15.77:0/164443780 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f47d0015600 con 0x7f47e00c8700 2024-04-25T01:22:30.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.643+0000 7f47d67fc640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f47c406be60 0x7f47c406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.644+0000 7f47d67fc640 1 -- 172.21.15.77:0/164443780 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f47d008bae0 con 0x7f47e00c8700 2024-04-25T01:22:30.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.644+0000 7f47e5d95640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f47c406be60 0x7f47c406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.644+0000 7f47e5d95640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f47c406be60 0x7f47c406e320 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f47d8035000 tx=0x7f47d802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.648+0000 7f47d67fc640 1 -- 172.21.15.77:0/164443780 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f47d0019030 con 0x7f47e00c8700 2024-04-25T01:22:30.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.822+0000 7f47e7ff9640 1 -- 172.21.15.77:0/164443780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f47e00cd990 con 0x7f47e00c8700 2024-04-25T01:22:30.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.822+0000 7f47d67fc640 1 -- 172.21.15.77:0/164443780 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f47d005a110 con 0x7f47e00c8700 2024-04-25T01:22:30.804 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 -- 172.21.15.77:0/164443780 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f47c406be60 msgr2=0x7f47c406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f47c406be60 0x7f47c406e320 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f47d8035000 tx=0x7f47d802f000 comp rx=0 tx=0).stop 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 -- 172.21.15.77:0/164443780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47e00c8700 msgr2=0x7f47e015d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47e00c8700 0x7f47e015d0f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f47d0011700 tx=0x7f47d0011bc0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 -- 172.21.15.77:0/164443780 shutdown_connections 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f47c406be60 0x7f47c406e320 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47e00c8700 0x7f47e015d0f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47e015d630 0x7f47e006e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 --2- 172.21.15.77:0/164443780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47e015b490 0x7f47e00c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 -- 172.21.15.77:0/164443780 >> 172.21.15.77:0/164443780 conn(0x7f47e00b1f00 msgr2=0x7f47e00ca980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 -- 172.21.15.77:0/164443780 shutdown_connections 2024-04-25T01:22:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.824+0000 7f47e7ff9640 1 -- 172.21.15.77:0/164443780 wait complete. 2024-04-25T01:22:30.819 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:30.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.935+0000 7f713b91b640 1 Processor -- start 2024-04-25T01:22:30.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.936+0000 7f713b91b640 1 -- start start 2024-04-25T01:22:30.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f713b91b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f713406ce30 0x7f713406d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f713b91b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f71340b30a0 0x7f71340b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f713b91b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f713415fad0 con 0x7f7134152f00 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f713b91b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f71340b54c0 con 0x7f71340b30a0 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f713b91b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f713415fdb0 con 0x7f713406ce30 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f7139eb8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f71340b30a0 0x7f71340b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f7139eb8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f71340b30a0 0x7f71340b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49074/0 (socket says 172.21.15.77:49074) 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f7139eb8640 1 -- 172.21.15.77:0/3535617001 learned_addr learned my addr 172.21.15.77:0/3535617001 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f7138eb6640 1 --2- 172.21.15.77:0/3535617001 >> v2:172.21.15.121:3300/0 conn(0x7f713406ce30 0x7f713406d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f7139eb8640 1 -- 172.21.15.77:0/3535617001 >> v2:172.21.15.121:3300/0 conn(0x7f713406ce30 msgr2=0x7f713406d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f7139eb8640 1 --2- 172.21.15.77:0/3535617001 >> v2:172.21.15.121:3300/0 conn(0x7f713406ce30 0x7f713406d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f7139eb8640 1 -- 172.21.15.77:0/3535617001 >> v1:172.21.15.121:6789/0 conn(0x7f7134152f00 legacy=0x7f71340b4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.937+0000 7f7139eb8640 1 -- 172.21.15.77:0/3535617001 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7134160090 con 0x7f71340b30a0 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.938+0000 7f7139eb8640 1 --2- 172.21.15.77:0/3535617001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71340b30a0 0x7f71340b34d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7134073390 tx=0x7f712c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f7f94ed6423464 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.938+0000 7f712bfff640 1 -- 172.21.15.77:0/3535617001 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c036070 con 0x7f71340b30a0 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.938+0000 7f712bfff640 1 -- 172.21.15.77:0/3535617001 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f712c031600 con 0x7f71340b30a0 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.938+0000 7f712bfff640 1 -- 172.21.15.77:0/3535617001 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c0319d0 con 0x7f71340b30a0 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.938+0000 7f713b91b640 1 -- 172.21.15.77:0/3535617001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71340b30a0 msgr2=0x7f71340b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.938+0000 7f713b91b640 1 --2- 172.21.15.77:0/3535617001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71340b30a0 0x7f71340b34d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7134073390 tx=0x7f712c02a6a0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.938+0000 7f713b91b640 1 -- 172.21.15.77:0/3535617001 shutdown_connections 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.938+0000 7f713b91b640 1 --2- 172.21.15.77:0/3535617001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71340b30a0 0x7f71340b34d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.938+0000 7f713b91b640 1 --2- 172.21.15.77:0/3535617001 >> v2:172.21.15.121:3300/0 conn(0x7f713406ce30 0x7f713406d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.939+0000 7f713b91b640 1 -- 172.21.15.77:0/3535617001 >> 172.21.15.77:0/3535617001 conn(0x7f71340b1c60 msgr2=0x7f71340b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.939+0000 7f713b91b640 1 -- 172.21.15.77:0/3535617001 shutdown_connections 2024-04-25T01:22:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.939+0000 7f713b91b640 1 -- 172.21.15.77:0/3535617001 wait complete. 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.940+0000 7f713b91b640 1 Processor -- start 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.940+0000 7f713b91b640 1 -- start start 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.940+0000 7f713b91b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713406ce30 0x7f71341410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.940+0000 7f713b91b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71340b30a0 0x7f71340b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.940+0000 7f713b91b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7134152f00 0x7f71340b1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.940+0000 7f713b91b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f713415d130 con 0x7f71340b30a0 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.940+0000 7f713b91b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f71340ce4a0 con 0x7f7134152f00 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.940+0000 7f713b91b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f713415d2b0 con 0x7f713406ce30 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7138eb6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71340b30a0 0x7f71340b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7138eb6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71340b30a0 0x7f71340b0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49080/0 (socket says 172.21.15.77:49080) 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7138eb6640 1 -- 172.21.15.77:0/2001510917 learned_addr learned my addr 172.21.15.77:0/2001510917 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f71396b7640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713406ce30 0x7f71341410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7139eb8640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7134152f00 0x7f71340b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7138eb6640 1 -- 172.21.15.77:0/2001510917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713406ce30 msgr2=0x7f71341410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7138eb6640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713406ce30 0x7f71341410b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7138eb6640 1 -- 172.21.15.77:0/2001510917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7134152f00 msgr2=0x7f71340b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7138eb6640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7134152f00 0x7f71340b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7138eb6640 1 -- 172.21.15.77:0/2001510917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7134159b60 con 0x7f71340b30a0 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7138eb6640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71340b30a0 0x7f71340b0d20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f712400e8a0 tx=0x7f712400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7129ffb640 1 -- 172.21.15.77:0/2001510917 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7124017070 con 0x7f71340b30a0 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7129ffb640 1 -- 172.21.15.77:0/2001510917 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f712400c940 con 0x7f71340b30a0 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f713b91b640 1 -- 172.21.15.77:0/2001510917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7134141730 con 0x7f71340b30a0 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f7129ffb640 1 -- 172.21.15.77:0/2001510917 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712400cd30 con 0x7f71340b30a0 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.941+0000 7f713b91b640 1 -- 172.21.15.77:0/2001510917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7134141cf0 con 0x7f71340b30a0 2024-04-25T01:22:30.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.942+0000 7f713b91b640 1 -- 172.21.15.77:0/2001510917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f713413f220 con 0x7f71340b30a0 2024-04-25T01:22:30.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.943+0000 7f7129ffb640 1 -- 172.21.15.77:0/2001510917 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f712401a070 con 0x7f71340b30a0 2024-04-25T01:22:30.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.943+0000 7f7129ffb640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f711806be30 0x7f711806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.943+0000 7f7129ffb640 1 -- 172.21.15.77:0/2001510917 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f712408b180 con 0x7f71340b30a0 2024-04-25T01:22:30.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.943+0000 7f71396b7640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f711806be30 0x7f711806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.944+0000 7f71396b7640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f711806be30 0x7f711806e2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7130001640 tx=0x7f7130009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:30.947+0000 7f7129ffb640 1 -- 172.21.15.77:0/2001510917 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f713413f220 con 0x7f71340b30a0 2024-04-25T01:22:31.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.118+0000 7f713b91b640 1 -- 172.21.15.77:0/2001510917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7134068fb0 con 0x7f71340b30a0 2024-04-25T01:22:31.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.119+0000 7f7129ffb640 1 -- 172.21.15.77:0/2001510917 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f7124059680 con 0x7f71340b30a0 2024-04-25T01:22:31.100 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 -- 172.21.15.77:0/2001510917 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f711806be30 msgr2=0x7f711806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f711806be30 0x7f711806e2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7130001640 tx=0x7f7130009040 comp rx=0 tx=0).stop 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 -- 172.21.15.77:0/2001510917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71340b30a0 msgr2=0x7f71340b0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71340b30a0 0x7f71340b0d20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f712400e8a0 tx=0x7f712400ed60 comp rx=0 tx=0).stop 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 -- 172.21.15.77:0/2001510917 shutdown_connections 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f711806be30 0x7f711806e2f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71340b30a0 0x7f71340b0d20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7134152f00 0x7f71340b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 --2- 172.21.15.77:0/2001510917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713406ce30 0x7f71341410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 -- 172.21.15.77:0/2001510917 >> 172.21.15.77:0/2001510917 conn(0x7f71340b1c60 msgr2=0x7f713415e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 -- 172.21.15.77:0/2001510917 shutdown_connections 2024-04-25T01:22:31.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.121+0000 7f713b91b640 1 -- 172.21.15.77:0/2001510917 wait complete. 2024-04-25T01:22:31.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:31.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.230+0000 7fe31755a640 1 Processor -- start 2024-04-25T01:22:31.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.231+0000 7fe31755a640 1 -- start start 2024-04-25T01:22:31.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.231+0000 7fe31755a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe3100d23d0 0x7fe3100d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.231+0000 7fe31755a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe3100d27b0 0x7fe3100d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.231+0000 7fe31755a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3100b9380 0x7fe3100b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.231+0000 7fe31755a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fe3100cea40 con 0x7fe3100d23d0 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.231+0000 7fe31755a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe3100b9d10 con 0x7fe3100d27b0 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.231+0000 7fe31755a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe3100cece0 con 0x7fe3100b9380 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.231+0000 7fe3152f6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe3100d23d0 0x7fe3100d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.232+0000 7fe3152f6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe3100d23d0 0x7fe3100d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49088/0 (socket says 172.21.15.77:49088) 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.232+0000 7fe3152f6640 1 -- 172.21.15.77:0/2102876120 learned_addr learned my addr 172.21.15.77:0/2102876120 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.232+0000 7fe314af5640 1 --2- 172.21.15.77:0/2102876120 >> v2:172.21.15.121:3300/0 conn(0x7fe3100d27b0 0x7fe3100d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.232+0000 7fe315af7640 1 --2- 172.21.15.77:0/2102876120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3100b9380 0x7fe3100b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.232+0000 7fe3152f6640 1 -- 172.21.15.77:0/2102876120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3100b9380 msgr2=0x7fe3100b97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.232+0000 7fe3152f6640 1 --2- 172.21.15.77:0/2102876120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3100b9380 0x7fe3100b97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.232+0000 7fe3152f6640 1 -- 172.21.15.77:0/2102876120 >> v2:172.21.15.121:3300/0 conn(0x7fe3100d27b0 msgr2=0x7fe3100d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.232+0000 7fe3152f6640 1 --2- 172.21.15.77:0/2102876120 >> v2:172.21.15.121:3300/0 conn(0x7fe3100d27b0 0x7fe3100d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.232+0000 7fe3152f6640 1 -- 172.21.15.77:0/2102876120 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3100cefc0 con 0x7fe3100d23d0 2024-04-25T01:22:31.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe3152f6640 1 --2- 172.21.15.77:0/2102876120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3100d23d0 0x7fe3100d4920 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe310067900 tx=0x7fe30c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2f6c2ed613bc49 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe307fff640 1 -- 172.21.15.77:0/2102876120 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c037070 con 0x7fe3100d23d0 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe307fff640 1 -- 172.21.15.77:0/2102876120 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe30c032440 con 0x7fe3100d23d0 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe307fff640 1 -- 172.21.15.77:0/2102876120 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c032810 con 0x7fe3100d23d0 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe31755a640 1 -- 172.21.15.77:0/2102876120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3100d23d0 msgr2=0x7fe3100d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe31755a640 1 --2- 172.21.15.77:0/2102876120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3100d23d0 0x7fe3100d4920 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe310067900 tx=0x7fe30c02a270 comp rx=0 tx=0).stop 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe31755a640 1 -- 172.21.15.77:0/2102876120 shutdown_connections 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe31755a640 1 --2- 172.21.15.77:0/2102876120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3100b9380 0x7fe3100b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe31755a640 1 --2- 172.21.15.77:0/2102876120 >> v2:172.21.15.121:3300/0 conn(0x7fe3100d27b0 0x7fe3100d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe31755a640 1 --2- 172.21.15.77:0/2102876120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3100d23d0 0x7fe3100d4920 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe31755a640 1 -- 172.21.15.77:0/2102876120 >> 172.21.15.77:0/2102876120 conn(0x7fe3100d3e30 msgr2=0x7fe3100d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe31755a640 1 -- 172.21.15.77:0/2102876120 shutdown_connections 2024-04-25T01:22:31.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.233+0000 7fe31755a640 1 -- 172.21.15.77:0/2102876120 wait complete. 2024-04-25T01:22:31.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.234+0000 7fe31755a640 1 Processor -- start 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.234+0000 7fe31755a640 1 -- start start 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe31755a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3100b9380 0x7fe31014e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe31755a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3100d23d0 0x7fe31015c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe31755a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3100d27b0 0x7fe3100b75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe31755a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe3100cf860 con 0x7fe3100d27b0 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe31755a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe3100cf580 con 0x7fe3100d23d0 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe31755a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe3100cfd80 con 0x7fe3100b9380 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe315af7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3100d27b0 0x7fe3100b75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe315af7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3100d27b0 0x7fe3100b75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49098/0 (socket says 172.21.15.77:49098) 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe315af7640 1 -- 172.21.15.77:0/4127518101 learned_addr learned my addr 172.21.15.77:0/4127518101 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe3152f6640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3100b9380 0x7fe31014e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe315af7640 1 -- 172.21.15.77:0/4127518101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3100b9380 msgr2=0x7fe31014e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe314af5640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3100d23d0 0x7fe31015c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.235+0000 7fe315af7640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3100b9380 0x7fe31014e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.236+0000 7fe315af7640 1 -- 172.21.15.77:0/4127518101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3100d23d0 msgr2=0x7fe31015c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.236+0000 7fe315af7640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3100d23d0 0x7fe31015c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.236+0000 7fe315af7640 1 -- 172.21.15.77:0/4127518101 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31015cfa0 con 0x7fe3100d27b0 2024-04-25T01:22:31.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.236+0000 7fe315af7640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3100d27b0 0x7fe3100b75e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe308007e60 tx=0x7fe30800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:31.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.236+0000 7fe305ffb640 1 -- 172.21.15.77:0/4127518101 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308014620 con 0x7fe3100d27b0 2024-04-25T01:22:31.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.236+0000 7fe305ffb640 1 -- 172.21.15.77:0/4127518101 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe308019070 con 0x7fe3100d27b0 2024-04-25T01:22:31.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.236+0000 7fe31755a640 1 -- 172.21.15.77:0/4127518101 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31015c9e0 con 0x7fe3100d27b0 2024-04-25T01:22:31.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.236+0000 7fe305ffb640 1 -- 172.21.15.77:0/4127518101 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308014a20 con 0x7fe3100d27b0 2024-04-25T01:22:31.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.236+0000 7fe31755a640 1 -- 172.21.15.77:0/4127518101 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe310159910 con 0x7fe3100d27b0 2024-04-25T01:22:31.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.237+0000 7fe31755a640 1 -- 172.21.15.77:0/4127518101 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe310068fb0 con 0x7fe3100d27b0 2024-04-25T01:22:31.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.237+0000 7fe305ffb640 1 -- 172.21.15.77:0/4127518101 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe308014bc0 con 0x7fe3100d27b0 2024-04-25T01:22:31.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.238+0000 7fe305ffb640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe2f406be30 0x7fe2f406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.238+0000 7fe3152f6640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe2f406be30 0x7fe2f406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.238+0000 7fe305ffb640 1 -- 172.21.15.77:0/4127518101 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fe30808bb40 con 0x7fe3100d27b0 2024-04-25T01:22:31.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.238+0000 7fe3152f6640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe2f406be30 0x7fe2f406e2f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe30c034000 tx=0x7fe30c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:31.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.242+0000 7fe305ffb640 1 -- 172.21.15.77:0/4127518101 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe30805a0e0 con 0x7fe3100d27b0 2024-04-25T01:22:31.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.415+0000 7fe31755a640 1 -- 172.21.15.77:0/4127518101 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe310159ae0 con 0x7fe3100d27b0 2024-04-25T01:22:31.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.416+0000 7fe305ffb640 1 -- 172.21.15.77:0/4127518101 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fe3080599e0 con 0x7fe3100d27b0 2024-04-25T01:22:31.397 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 -- 172.21.15.77:0/4127518101 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe2f406be30 msgr2=0x7fe2f406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe2f406be30 0x7fe2f406e2f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe30c034000 tx=0x7fe30c02e000 comp rx=0 tx=0).stop 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 -- 172.21.15.77:0/4127518101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3100d27b0 msgr2=0x7fe3100b75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3100d27b0 0x7fe3100b75e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe308007e60 tx=0x7fe30800e4b0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 -- 172.21.15.77:0/4127518101 shutdown_connections 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe2f406be30 0x7fe2f406e2f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3100d27b0 0x7fe3100b75e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3100d23d0 0x7fe31015c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 --2- 172.21.15.77:0/4127518101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3100b9380 0x7fe31014e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 -- 172.21.15.77:0/4127518101 >> 172.21.15.77:0/4127518101 conn(0x7fe3100d3e30 msgr2=0x7fe31006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 -- 172.21.15.77:0/4127518101 shutdown_connections 2024-04-25T01:22:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.418+0000 7fe31755a640 1 -- 172.21.15.77:0/4127518101 wait complete. 2024-04-25T01:22:31.410 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.528+0000 7fd4e84bd640 1 Processor -- start 2024-04-25T01:22:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.528+0000 7fd4e84bd640 1 -- start start 2024-04-25T01:22:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e84bd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd4e00af8c0 0x7fd4e00cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e84bd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd4e00b5060 0x7fd4e006b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e84bd640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4e00b0200 con 0x7fd4e006b660 2024-04-25T01:22:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e84bd640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fd4e006bad0 con 0x7fd4e00b5060 2024-04-25T01:22:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e84bd640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd4e00b5840 con 0x7fd4e00af8c0 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e5a58640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd4e00b5060 0x7fd4e006b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e6a5a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd4e006b660 0x7fd4e015d940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57444/0 (socket says 172.21.15.77:57444) 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e6a5a640 1 -- 172.21.15.77:0/4085302929 learned_addr learned my addr 172.21.15.77:0/4085302929 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e5a58640 1 -- 172.21.15.77:0/4085302929 >> v2:172.21.15.121:3300/0 conn(0x7fd4e00af8c0 msgr2=0x7fd4e00cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e5a58640 1 --2- 172.21.15.77:0/4085302929 >> v2:172.21.15.121:3300/0 conn(0x7fd4e00af8c0 0x7fd4e00cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e5a58640 1 -- 172.21.15.77:0/4085302929 >> v1:172.21.15.77:6789/0 conn(0x7fd4e006b660 legacy=0x7fd4e015d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e5257640 1 -- 172.21.15.77:0/4085302929 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3844586850 0 0) 0x7fd4e00b0200 con 0x7fd4e006b660 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.529+0000 7fd4e5a58640 1 -- 172.21.15.77:0/4085302929 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e00b5b20 con 0x7fd4e00b5060 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e5a58640 1 --2- 172.21.15.77:0/4085302929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e00b5060 0x7fd4e006b120 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0073980 tx=0x7fd4d002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523cf0c5f4d44630 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e5257640 1 -- 172.21.15.77:0/4085302929 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0037070 con 0x7fd4e00b5060 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e5257640 1 -- 172.21.15.77:0/4085302929 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d0032440 con 0x7fd4e00b5060 2024-04-25T01:22:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e5257640 1 -- 172.21.15.77:0/4085302929 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0032810 con 0x7fd4e00b5060 2024-04-25T01:22:31.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e84bd640 1 -- 172.21.15.77:0/4085302929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e00b5060 msgr2=0x7fd4e006b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e84bd640 1 --2- 172.21.15.77:0/4085302929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e00b5060 0x7fd4e006b120 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0073980 tx=0x7fd4d002a1a0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e84bd640 1 -- 172.21.15.77:0/4085302929 shutdown_connections 2024-04-25T01:22:31.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e84bd640 1 --2- 172.21.15.77:0/4085302929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e00b5060 0x7fd4e006b120 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e84bd640 1 --2- 172.21.15.77:0/4085302929 >> v2:172.21.15.121:3300/0 conn(0x7fd4e00af8c0 0x7fd4e00cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e84bd640 1 -- 172.21.15.77:0/4085302929 >> 172.21.15.77:0/4085302929 conn(0x7fd4e00b1f00 msgr2=0x7fd4e00b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e84bd640 1 -- 172.21.15.77:0/4085302929 shutdown_connections 2024-04-25T01:22:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.530+0000 7fd4e84bd640 1 -- 172.21.15.77:0/4085302929 wait complete. 2024-04-25T01:22:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.532+0000 7fd4e84bd640 1 Processor -- start 2024-04-25T01:22:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.532+0000 7fd4e84bd640 1 -- start start 2024-04-25T01:22:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.532+0000 7fd4e84bd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4e006b660 0x7fd4e0151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.532+0000 7fd4e84bd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4e00af8c0 0x7fd4e00c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.532+0000 7fd4e84bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e00b5060 0x7fd4e01f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.532+0000 7fd4e84bd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e00b5510 con 0x7fd4e00b5060 2024-04-25T01:22:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.532+0000 7fd4e84bd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e00b60e0 con 0x7fd4e00af8c0 2024-04-25T01:22:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.532+0000 7fd4e84bd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd4e00afca0 con 0x7fd4e006b660 2024-04-25T01:22:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.532+0000 7fd4e6a5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e00b5060 0x7fd4e01f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.532+0000 7fd4e6a5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e00b5060 0x7fd4e01f70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49120/0 (socket says 172.21.15.77:49120) 2024-04-25T01:22:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.532+0000 7fd4e6a5a640 1 -- 172.21.15.77:0/3467614192 learned_addr learned my addr 172.21.15.77:0/3467614192 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4e6259640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4e006b660 0x7fd4e0151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4e6a5a640 1 -- 172.21.15.77:0/3467614192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4e006b660 msgr2=0x7fd4e0151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4e5a58640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4e00af8c0 0x7fd4e00c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4e6a5a640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4e006b660 0x7fd4e0151900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4e6a5a640 1 -- 172.21.15.77:0/3467614192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4e00af8c0 msgr2=0x7fd4e00c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4e6a5a640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4e00af8c0 0x7fd4e00c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4e6a5a640 1 -- 172.21.15.77:0/3467614192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e00c9e30 con 0x7fd4e00b5060 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4e6a5a640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e00b5060 0x7fd4e01f70c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800c480 tx=0x7fd4d800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4d6ffd640 1 -- 172.21.15.77:0/3467614192 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8017070 con 0x7fd4e00b5060 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4d6ffd640 1 -- 172.21.15.77:0/3467614192 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d8005d00 con 0x7fd4e00b5060 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4e84bd640 1 -- 172.21.15.77:0/3467614192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4e00ca0d0 con 0x7fd4e00b5060 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4d6ffd640 1 -- 172.21.15.77:0/3467614192 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80072b0 con 0x7fd4e00b5060 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.533+0000 7fd4e84bd640 1 -- 172.21.15.77:0/3467614192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4e01f7600 con 0x7fd4e00b5060 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.534+0000 7fd4e84bd640 1 -- 172.21.15.77:0/3467614192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4e0068fb0 con 0x7fd4e00b5060 2024-04-25T01:22:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.534+0000 7fd4d6ffd640 1 -- 172.21.15.77:0/3467614192 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd4d801a070 con 0x7fd4e00b5060 2024-04-25T01:22:31.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.535+0000 7fd4d6ffd640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd4c406be30 0x7fd4c406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.535+0000 7fd4e6259640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd4c406be30 0x7fd4c406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.535+0000 7fd4d6ffd640 1 -- 172.21.15.77:0/3467614192 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fd4d808a610 con 0x7fd4e00b5060 2024-04-25T01:22:31.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.535+0000 7fd4e6259640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd4c406be30 0x7fd4c406e2f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc007dc0 tx=0x7fd4dc0072f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:31.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.539+0000 7fd4d6ffd640 1 -- 172.21.15.77:0/3467614192 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd4d8058c40 con 0x7fd4e00b5060 2024-04-25T01:22:31.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.711+0000 7fd4e84bd640 1 -- 172.21.15.77:0/3467614192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd4e006eae0 con 0x7fd4e00b5060 2024-04-25T01:22:31.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.711+0000 7fd4d6ffd640 1 -- 172.21.15.77:0/3467614192 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fd4d8058540 con 0x7fd4e00b5060 2024-04-25T01:22:31.693 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:31.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 -- 172.21.15.77:0/3467614192 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd4c406be30 msgr2=0x7fd4c406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd4c406be30 0x7fd4c406e2f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc007dc0 tx=0x7fd4dc0072f0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 -- 172.21.15.77:0/3467614192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e00b5060 msgr2=0x7fd4e01f70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e00b5060 0x7fd4e01f70c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800c480 tx=0x7fd4d800c940 comp rx=0 tx=0).stop 2024-04-25T01:22:31.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 -- 172.21.15.77:0/3467614192 shutdown_connections 2024-04-25T01:22:31.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd4c406be30 0x7fd4c406e2f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e00b5060 0x7fd4e01f70c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4e00af8c0 0x7fd4e00c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 --2- 172.21.15.77:0/3467614192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4e006b660 0x7fd4e0151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 -- 172.21.15.77:0/3467614192 >> 172.21.15.77:0/3467614192 conn(0x7fd4e00b1f00 msgr2=0x7fd4e00d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:31.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 -- 172.21.15.77:0/3467614192 shutdown_connections 2024-04-25T01:22:31.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.713+0000 7fd4e84bd640 1 -- 172.21.15.77:0/3467614192 wait complete. 2024-04-25T01:22:31.733 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace_repair (tasks.cephfs.test_auto_repair.TestMDSAutoRepair) MDS should verify/fix backtrace on fetch dirfrag 2024-04-25T01:22:31.733 INFO:tasks.cephfs_test_runner:test_backtrace_repair (tasks.cephfs.test_auto_repair.TestMDSAutoRepair) 2024-04-25T01:22:31.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:31.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.849+0000 7fe8dcf6a640 1 Processor -- start 2024-04-25T01:22:31.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.849+0000 7fe8dcf6a640 1 -- start start 2024-04-25T01:22:31.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8dcf6a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8d806e480 0x7fe8d806e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8dcf6a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8d815bf70 0x7fe8d80cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8dcf6a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8d80cd1f0 con 0x7fe8d80cf8f0 2024-04-25T01:22:31.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8dcf6a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fe8d80d0dd0 con 0x7fe8d806e480 2024-04-25T01:22:31.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8dcf6a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe8d80ce670 con 0x7fe8d815bf70 2024-04-25T01:22:31.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8d659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8d806e480 0x7fe8d806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8d659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8d806e480 0x7fe8d806e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49132/0 (socket says 172.21.15.77:49132) 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8d6d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe8d80cf8f0 0x7fe8d815f8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57452/0 (socket says 172.21.15.77:57452) 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8d659c640 1 -- 172.21.15.77:0/1140755099 learned_addr learned my addr 172.21.15.77:0/1140755099 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8cffff640 1 --2- 172.21.15.77:0/1140755099 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8d815bf70 0x7fe8d80cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8d659c640 1 -- 172.21.15.77:0/1140755099 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8d815bf70 msgr2=0x7fe8d80cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8d659c640 1 --2- 172.21.15.77:0/1140755099 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8d815bf70 0x7fe8d80cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8d659c640 1 -- 172.21.15.77:0/1140755099 >> v1:172.21.15.77:6789/0 conn(0x7fe8d80cf8f0 legacy=0x7fe8d815f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8d659c640 1 -- 172.21.15.77:0/1140755099 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d806eda0 con 0x7fe8d806e480 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.850+0000 7fe8d5d9b640 1 -- 172.21.15.77:0/1140755099 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2189001695 0 0) 0x7fe8d80cd1f0 con 0x7fe8d80cf8f0 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.851+0000 7fe8d659c640 1 --2- 172.21.15.77:0/1140755099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d806e480 0x7fe8d806e860 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009870 tx=0x7fe8c802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6adff3947e631c4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.851+0000 7fe8d5d9b640 1 -- 172.21.15.77:0/1140755099 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8037070 con 0x7fe8d806e480 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.851+0000 7fe8d5d9b640 1 -- 172.21.15.77:0/1140755099 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8c8032440 con 0x7fe8d806e480 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.851+0000 7fe8d5d9b640 1 -- 172.21.15.77:0/1140755099 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8032810 con 0x7fe8d806e480 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.851+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/1140755099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d806e480 msgr2=0x7fe8d806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.851+0000 7fe8dcf6a640 1 --2- 172.21.15.77:0/1140755099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d806e480 0x7fe8d806e860 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009870 tx=0x7fe8c802a1c0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.852+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/1140755099 shutdown_connections 2024-04-25T01:22:31.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.852+0000 7fe8dcf6a640 1 --2- 172.21.15.77:0/1140755099 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8d815bf70 0x7fe8d80cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.852+0000 7fe8dcf6a640 1 --2- 172.21.15.77:0/1140755099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d806e480 0x7fe8d806e860 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.852+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/1140755099 >> 172.21.15.77:0/1140755099 conn(0x7fe8d80cff90 msgr2=0x7fe8d80d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:31.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.852+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/1140755099 shutdown_connections 2024-04-25T01:22:31.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.852+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/1140755099 wait complete. 2024-04-25T01:22:31.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.853+0000 7fe8dcf6a640 1 Processor -- start 2024-04-25T01:22:31.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.853+0000 7fe8dcf6a640 1 -- start start 2024-04-25T01:22:31.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8dcf6a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8d806e480 0x7fe8d8158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8dcf6a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8d80cf8f0 0x7fe8d8159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8dcf6a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d815bf70 0x7fe8d8146540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8dcf6a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d80cebb0 con 0x7fe8d815bf70 2024-04-25T01:22:31.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8dcf6a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d80ce8d0 con 0x7fe8d806e480 2024-04-25T01:22:31.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8dcf6a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe8d80cee90 con 0x7fe8d80cf8f0 2024-04-25T01:22:31.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8d6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d815bf70 0x7fe8d8146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8d6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d815bf70 0x7fe8d8146540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49140/0 (socket says 172.21.15.77:49140) 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8d6d9d640 1 -- 172.21.15.77:0/3370408603 learned_addr learned my addr 172.21.15.77:0/3370408603 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8cffff640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8d80cf8f0 0x7fe8d8159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8d659c640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8d806e480 0x7fe8d8158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8d6d9d640 1 -- 172.21.15.77:0/3370408603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8d80cf8f0 msgr2=0x7fe8d8159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8d6d9d640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8d80cf8f0 0x7fe8d8159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8d6d9d640 1 -- 172.21.15.77:0/3370408603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8d806e480 msgr2=0x7fe8d8158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8d6d9d640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8d806e480 0x7fe8d8158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8d6d9d640 1 -- 172.21.15.77:0/3370408603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d80ae890 con 0x7fe8d815bf70 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8cffff640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8d80cf8f0 0x7fe8d8159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8d6d9d640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d815bf70 0x7fe8d8146540 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000c760 tx=0x7fe8d000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8ceffd640 1 -- 172.21.15.77:0/3370408603 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0018070 con 0x7fe8d815bf70 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8ceffd640 1 -- 172.21.15.77:0/3370408603 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d000a640 con 0x7fe8d815bf70 2024-04-25T01:22:31.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/3370408603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d80aeb70 con 0x7fe8d815bf70 2024-04-25T01:22:31.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8ceffd640 1 -- 172.21.15.77:0/3370408603 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d000aa30 con 0x7fe8d815bf70 2024-04-25T01:22:31.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.854+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/3370408603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d8071700 con 0x7fe8d815bf70 2024-04-25T01:22:31.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.855+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/3370408603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8d8071900 con 0x7fe8d815bf70 2024-04-25T01:22:31.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.856+0000 7fe8ceffd640 1 -- 172.21.15.77:0/3370408603 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe8d001b070 con 0x7fe8d815bf70 2024-04-25T01:22:31.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.856+0000 7fe8ceffd640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe8b806be30 0x7fe8b806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:31.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.856+0000 7fe8ceffd640 1 -- 172.21.15.77:0/3370408603 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fe8d008a880 con 0x7fe8d815bf70 2024-04-25T01:22:31.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.856+0000 7fe8d659c640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe8b806be30 0x7fe8b806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:31.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.857+0000 7fe8d659c640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe8b806be30 0x7fe8b806e2f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8034000 tx=0x7fe8c802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:31.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:31.861+0000 7fe8ceffd640 1 -- 172.21.15.77:0/3370408603 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe8d8071900 con 0x7fe8d815bf70 2024-04-25T01:22:32.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.034+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/3370408603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe8d8068fb0 con 0x7fe8d815bf70 2024-04-25T01:22:32.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.035+0000 7fe8ceffd640 1 -- 172.21.15.77:0/3370408603 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18922 (secure 0 0 0) 0x7fe8d0058e20 con 0x7fe8d815bf70 2024-04-25T01:22:32.017 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:32.017 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:26.287128+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:32.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/3370408603 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe8b806be30 msgr2=0x7fe8b806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe8b806be30 0x7fe8b806e2f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8034000 tx=0x7fe8c802e000 comp rx=0 tx=0).stop 2024-04-25T01:22:32.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/3370408603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d815bf70 msgr2=0x7fe8d8146540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d815bf70 0x7fe8d8146540 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000c760 tx=0x7fe8d000cc20 comp rx=0 tx=0).stop 2024-04-25T01:22:32.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/3370408603 shutdown_connections 2024-04-25T01:22:32.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe8b806be30 0x7fe8b806e2f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d815bf70 0x7fe8d8146540 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8d80cf8f0 0x7fe8d8159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 --2- 172.21.15.77:0/3370408603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8d806e480 0x7fe8d8158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/3370408603 >> 172.21.15.77:0/3370408603 conn(0x7fe8d80cff90 msgr2=0x7fe8d80b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:32.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/3370408603 shutdown_connections 2024-04-25T01:22:32.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.037+0000 7fe8dcf6a640 1 -- 172.21.15.77:0/3370408603 wait complete. 2024-04-25T01:22:32.030 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:32.031 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:32.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.148+0000 7fb31c570640 1 Processor -- start 2024-04-25T01:22:32.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.148+0000 7fb31c570640 1 -- start start 2024-04-25T01:22:32.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.149+0000 7fb31c570640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb31415e460 0x7fb3140d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.149+0000 7fb31c570640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3140d09a0 0x7fb3140ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.149+0000 7fb31c570640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3140d0de0 con 0x7fb3140cea10 2024-04-25T01:22:32.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.149+0000 7fb31c570640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fb31406c230 con 0x7fb3140d09a0 2024-04-25T01:22:32.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.149+0000 7fb31c570640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb3140cca90 con 0x7fb31415e460 2024-04-25T01:22:32.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.149+0000 7fb31ab0d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb3140cea10 0x7fb31406f050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57460/0 (socket says 172.21.15.77:57460) 2024-04-25T01:22:32.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.149+0000 7fb319b0b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3140d09a0 0x7fb3140ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.149+0000 7fb31ab0d640 1 -- 172.21.15.77:0/2799675520 learned_addr learned my addr 172.21.15.77:0/2799675520 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:32.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.149+0000 7fb31a30c640 1 --2- 172.21.15.77:0/2799675520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb31415e460 0x7fb3140d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.150+0000 7fb31930a640 1 -- 172.21.15.77:0/2799675520 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2401661803 0 0) 0x7fb3140d0de0 con 0x7fb3140cea10 2024-04-25T01:22:32.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.150+0000 7fb31930a640 1 -- 172.21.15.77:0/2799675520 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2fc003520 con 0x7fb3140cea10 2024-04-25T01:22:32.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.150+0000 7fb319b0b640 1 -- 172.21.15.77:0/2799675520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb31415e460 msgr2=0x7fb3140d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:32.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.150+0000 7fb319b0b640 1 --2- 172.21.15.77:0/2799675520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb31415e460 0x7fb3140d2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.150+0000 7fb319b0b640 1 -- 172.21.15.77:0/2799675520 >> v1:172.21.15.77:6789/0 conn(0x7fb3140cea10 legacy=0x7fb31406f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.150+0000 7fb319b0b640 1 -- 172.21.15.77:0/2799675520 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb314070280 con 0x7fb3140d09a0 2024-04-25T01:22:32.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.150+0000 7fb319b0b640 1 --2- 172.21.15.77:0/2799675520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3140d09a0 0x7fb3140ce4d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb3080097a0 tx=0x7fb30802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761bd5d462794e52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.150+0000 7fb31930a640 1 -- 172.21.15.77:0/2799675520 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308037070 con 0x7fb3140d09a0 2024-04-25T01:22:32.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.150+0000 7fb31930a640 1 -- 172.21.15.77:0/2799675520 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb308032440 con 0x7fb3140d09a0 2024-04-25T01:22:32.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.150+0000 7fb31930a640 1 -- 172.21.15.77:0/2799675520 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308032810 con 0x7fb3140d09a0 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.151+0000 7fb31c570640 1 -- 172.21.15.77:0/2799675520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3140d09a0 msgr2=0x7fb3140ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.151+0000 7fb31c570640 1 --2- 172.21.15.77:0/2799675520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3140d09a0 0x7fb3140ce4d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb3080097a0 tx=0x7fb30802a210 comp rx=0 tx=0).stop 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.151+0000 7fb31c570640 1 -- 172.21.15.77:0/2799675520 shutdown_connections 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.151+0000 7fb31c570640 1 --2- 172.21.15.77:0/2799675520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3140d09a0 0x7fb3140ce4d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.151+0000 7fb31c570640 1 --2- 172.21.15.77:0/2799675520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb31415e460 0x7fb3140d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.151+0000 7fb31c570640 1 -- 172.21.15.77:0/2799675520 >> 172.21.15.77:0/2799675520 conn(0x7fb3140d1c50 msgr2=0x7fb3140d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.151+0000 7fb31c570640 1 -- 172.21.15.77:0/2799675520 shutdown_connections 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.151+0000 7fb31c570640 1 -- 172.21.15.77:0/2799675520 wait complete. 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.152+0000 7fb31c570640 1 Processor -- start 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.152+0000 7fb31c570640 1 -- start start 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31c570640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3140cea10 0x7fb31414c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31c570640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3140d09a0 0x7fb3140cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31c570640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415e460 0x7fb3140c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31c570640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb3140cc720 con 0x7fb31415e460 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31c570640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb3140cc440 con 0x7fb3140cea10 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31c570640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb3140cdf30 con 0x7fb3140d09a0 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31ab0d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415e460 0x7fb3140c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31ab0d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415e460 0x7fb3140c7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49152/0 (socket says 172.21.15.77:49152) 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31ab0d640 1 -- 172.21.15.77:0/138150264 learned_addr learned my addr 172.21.15.77:0/138150264 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31ab0d640 1 -- 172.21.15.77:0/138150264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3140d09a0 msgr2=0x7fb3140cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31a30c640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3140cea10 0x7fb31414c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31ab0d640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3140d09a0 0x7fb3140cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31ab0d640 1 -- 172.21.15.77:0/138150264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3140cea10 msgr2=0x7fb31414c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31ab0d640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3140cea10 0x7fb31414c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31ab0d640 1 -- 172.21.15.77:0/138150264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31414c5b0 con 0x7fb31415e460 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.153+0000 7fb31ab0d640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415e460 0x7fb3140c7230 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00cb10 tx=0x7fb30c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.154+0000 7fb312ffd640 1 -- 172.21.15.77:0/138150264 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c017070 con 0x7fb31415e460 2024-04-25T01:22:32.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.154+0000 7fb312ffd640 1 -- 172.21.15.77:0/138150264 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb30c00ac70 con 0x7fb31415e460 2024-04-25T01:22:32.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.154+0000 7fb31c570640 1 -- 172.21.15.77:0/138150264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3140c5820 con 0x7fb31415e460 2024-04-25T01:22:32.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.154+0000 7fb312ffd640 1 -- 172.21.15.77:0/138150264 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c00ae60 con 0x7fb31415e460 2024-04-25T01:22:32.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.154+0000 7fb31c570640 1 -- 172.21.15.77:0/138150264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3140c5de0 con 0x7fb31415e460 2024-04-25T01:22:32.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.155+0000 7fb31c570640 1 -- 172.21.15.77:0/138150264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb314068fb0 con 0x7fb31415e460 2024-04-25T01:22:32.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.155+0000 7fb312ffd640 1 -- 172.21.15.77:0/138150264 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb30c01a070 con 0x7fb31415e460 2024-04-25T01:22:32.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.155+0000 7fb312ffd640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb30006be30 0x7fb30006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.156+0000 7fb312ffd640 1 -- 172.21.15.77:0/138150264 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fb30c08afe0 con 0x7fb31415e460 2024-04-25T01:22:32.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.156+0000 7fb31a30c640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb30006be30 0x7fb30006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.156+0000 7fb31a30c640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb30006be30 0x7fb30006e2f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb3140d08b0 tx=0x7fb304008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.160+0000 7fb312ffd640 1 -- 172.21.15.77:0/138150264 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb30c059610 con 0x7fb31415e460 2024-04-25T01:22:32.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.334+0000 7fb31c570640 1 -- 172.21.15.77:0/138150264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb31406d9a0 con 0x7fb31415e460 2024-04-25T01:22:32.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.334+0000 7fb312ffd640 1 -- 172.21.15.77:0/138150264 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fb30c058f10 con 0x7fb31415e460 2024-04-25T01:22:32.316 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:32.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.336+0000 7fb31c570640 1 -- 172.21.15.77:0/138150264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb30006be30 msgr2=0x7fb30006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.336+0000 7fb31c570640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb30006be30 0x7fb30006e2f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb3140d08b0 tx=0x7fb304008040 comp rx=0 tx=0).stop 2024-04-25T01:22:32.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.336+0000 7fb31c570640 1 -- 172.21.15.77:0/138150264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415e460 msgr2=0x7fb3140c7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.336+0000 7fb31c570640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415e460 0x7fb3140c7230 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00cb10 tx=0x7fb30c00a060 comp rx=0 tx=0).stop 2024-04-25T01:22:32.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.336+0000 7fb31c570640 1 -- 172.21.15.77:0/138150264 shutdown_connections 2024-04-25T01:22:32.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.336+0000 7fb31c570640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb30006be30 0x7fb30006e2f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.336+0000 7fb31c570640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415e460 0x7fb3140c7230 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.336+0000 7fb31c570640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3140d09a0 0x7fb3140cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.336+0000 7fb31c570640 1 --2- 172.21.15.77:0/138150264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3140cea10 0x7fb31414c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.336+0000 7fb31c570640 1 -- 172.21.15.77:0/138150264 >> 172.21.15.77:0/138150264 conn(0x7fb3140d1c50 msgr2=0x7fb3140b0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:32.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.337+0000 7fb31c570640 1 -- 172.21.15.77:0/138150264 shutdown_connections 2024-04-25T01:22:32.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.337+0000 7fb31c570640 1 -- 172.21.15.77:0/138150264 wait complete. 2024-04-25T01:22:32.329 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:32.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.447+0000 7f7a0a612640 1 Processor -- start 2024-04-25T01:22:32.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.447+0000 7f7a0a612640 1 -- start start 2024-04-25T01:22:32.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.448+0000 7f7a0a612640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7a040b47d0 0x7f7a0415b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.448+0000 7f7a0a612640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a040b37c0 0x7f7a0415ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.448+0000 7f7a0a612640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a0415bfa0 con 0x7f7a04158a70 2024-04-25T01:22:32.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.448+0000 7f7a0a612640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7a0415c210 con 0x7f7a040b47d0 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.448+0000 7f7a0a612640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a040b2ce0 con 0x7f7a040b37c0 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.448+0000 7f7a03fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7a04158a70 0x7f7a040b40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57464/0 (socket says 172.21.15.77:57464) 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.448+0000 7f7a03fff640 1 -- 172.21.15.77:0/2557168268 learned_addr learned my addr 172.21.15.77:0/2557168268 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3352272379 0 0) 0x7f7a0415bfa0 con 0x7f7a04158a70 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79ec003520 con 0x7f7a04158a70 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a08baf640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a040b37c0 0x7f7a0415ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a037fe640 1 --2- 172.21.15.77:0/2557168268 >> v2:172.21.15.121:3300/0 conn(0x7f7a040b47d0 0x7f7a0415b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2861091695 0 0) 0x7f79ec003520 con 0x7f7a04158a70 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a040b37c0 msgr2=0x7f7a0415ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a040b37c0 0x7f7a0415ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 >> v2:172.21.15.121:3300/0 conn(0x7f7a040b47d0 msgr2=0x7f7a0415b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:32.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 --2- 172.21.15.77:0/2557168268 >> v2:172.21.15.121:3300/0 conn(0x7f7a040b47d0 0x7f7a0415b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a040b2fc0 con 0x7f7a04158a70 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f79f4001670 con 0x7f7a04158a70 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a037fe640 1 --2- 172.21.15.77:0/2557168268 >> v2:172.21.15.121:3300/0 conn(0x7f7a040b47d0 0x7f7a0415b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 >> v1:172.21.15.77:6789/0 conn(0x7f7a04158a70 legacy=0x7f7a040b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a040b37c0 0x7f79ec0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79ec003f30 0x7f79ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79ec004d40 0x7f79ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0415c210 con 0x7f79ec003f30 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a040b2ce0 con 0x7f79ec004d40 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f79ec00dc20 con 0x7f7a040b37c0 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a037fe640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79ec003f30 0x7f79ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a03fff640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a040b37c0 0x7f79ec0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a08baf640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79ec004d40 0x7f79ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a037fe640 1 -- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a040b37c0 msgr2=0x7f79ec0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:32.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a037fe640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a040b37c0 0x7f79ec0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a037fe640 1 -- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79ec004d40 msgr2=0x7f79ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a037fe640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79ec004d40 0x7f79ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a037fe640 1 -- 172.21.15.77:0/2557168268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ec003520 con 0x7f79ec003f30 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a03fff640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a040b37c0 0x7f79ec0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.449+0000 7f7a08baf640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79ec004d40 0x7f79ec0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.450+0000 7f7a037fe640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79ec003f30 0x7f79ec004800 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f79f80098a0 tx=0x7f79ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8052c0e770a446f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.450+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f800f040 con 0x7f79ec003f30 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.450+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79f802ad20 con 0x7f79ec003f30 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.450+0000 7f7a02ffd640 1 -- 172.21.15.77:0/2557168268 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f8004250 con 0x7f79ec003f30 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.451+0000 7f7a0a612640 1 -- 172.21.15.77:0/2557168268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79ec003f30 msgr2=0x7f79ec004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.451+0000 7f7a0a612640 1 --2- 172.21.15.77:0/2557168268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79ec003f30 0x7f79ec004800 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f79f80098a0 tx=0x7f79ec005830 comp rx=0 tx=0).stop 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.451+0000 7f7a08baf640 1 -- 172.21.15.77:0/2557168268 reap_dead start 2024-04-25T01:22:32.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.451+0000 7f7a0a612640 1 -- 172.21.15.77:0/2557168268 shutdown_connections 2024-04-25T01:22:32.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.451+0000 7f7a0a612640 1 -- 172.21.15.77:0/2557168268 >> 172.21.15.77:0/2557168268 conn(0x7f7a0406b720 msgr2=0x7f7a040b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:32.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.451+0000 7f7a0a612640 1 -- 172.21.15.77:0/2557168268 shutdown_connections 2024-04-25T01:22:32.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.451+0000 7f7a0a612640 1 -- 172.21.15.77:0/2557168268 wait complete. 2024-04-25T01:22:32.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.452+0000 7f7a0a612640 1 Processor -- start 2024-04-25T01:22:32.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.452+0000 7f7a0a612640 1 -- start start 2024-04-25T01:22:32.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a0a612640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79ec003f30 0x7f7a04141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a0a612640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a0415e1c0 0x7f7a041f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a0a612640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a041f4230 0x7f7a040ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a0a612640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0415ba60 con 0x7f7a0415e1c0 2024-04-25T01:22:32.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a0a612640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a04158e50 con 0x7f7a041f4230 2024-04-25T01:22:32.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a0a612640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a0415bbe0 con 0x7f79ec003f30 2024-04-25T01:22:32.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a037fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a0415e1c0 0x7f7a041f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a037fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a0415e1c0 0x7f7a041f6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49168/0 (socket says 172.21.15.77:49168) 2024-04-25T01:22:32.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a037fe640 1 -- 172.21.15.77:0/3302495774 learned_addr learned my addr 172.21.15.77:0/3302495774 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:32.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a03fff640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79ec003f30 0x7f7a04141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a08baf640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a041f4230 0x7f7a040ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a037fe640 1 -- 172.21.15.77:0/3302495774 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79ec003f30 msgr2=0x7f7a04141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a037fe640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79ec003f30 0x7f7a04141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a037fe640 1 -- 172.21.15.77:0/3302495774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a041f4230 msgr2=0x7f7a040ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a037fe640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a041f4230 0x7f7a040ce1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a037fe640 1 -- 172.21.15.77:0/3302495774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a040cf190 con 0x7f7a0415e1c0 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a037fe640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a0415e1c0 0x7f7a041f6fd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f79f802a810 tx=0x7f79f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.453+0000 7f7a00ff9640 1 -- 172.21.15.77:0/3302495774 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f8041070 con 0x7f7a0415e1c0 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.454+0000 7f7a00ff9640 1 -- 172.21.15.77:0/3302495774 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79f802ab20 con 0x7f7a0415e1c0 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.454+0000 7f7a0a612640 1 -- 172.21.15.77:0/3302495774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a040cf470 con 0x7f7a0415e1c0 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.454+0000 7f7a00ff9640 1 -- 172.21.15.77:0/3302495774 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f803d430 con 0x7f7a0415e1c0 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.454+0000 7f7a0a612640 1 -- 172.21.15.77:0/3302495774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a040cad60 con 0x7f7a0415e1c0 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.455+0000 7f7a0a612640 1 -- 172.21.15.77:0/3302495774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a04068fb0 con 0x7f7a0415e1c0 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.455+0000 7f7a00ff9640 1 -- 172.21.15.77:0/3302495774 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f79f8044070 con 0x7f7a0415e1c0 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.455+0000 7f7a00ff9640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f79f006be60 0x7f79f006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.456+0000 7f7a00ff9640 1 -- 172.21.15.77:0/3302495774 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f79f80aba30 con 0x7f7a0415e1c0 2024-04-25T01:22:32.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.456+0000 7f7a03fff640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f79f006be60 0x7f79f006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.456+0000 7f7a03fff640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f79f006be60 0x7f79f006e320 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f79f4000fb0 tx=0x7f79f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.460+0000 7f7a00ff9640 1 -- 172.21.15.77:0/3302495774 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f79f8079fd0 con 0x7f7a0415e1c0 2024-04-25T01:22:32.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.631+0000 7f7a0a612640 1 -- 172.21.15.77:0/3302495774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7a040cf640 con 0x7f7a0415e1c0 2024-04-25T01:22:32.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.632+0000 7f7a00ff9640 1 -- 172.21.15.77:0/3302495774 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f79f802acf0 con 0x7f7a0415e1c0 2024-04-25T01:22:32.613 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:32.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.633+0000 7f7a0a612640 1 -- 172.21.15.77:0/3302495774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f79f006be60 msgr2=0x7f79f006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.633+0000 7f7a0a612640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f79f006be60 0x7f79f006e320 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f79f4000fb0 tx=0x7f79f4009040 comp rx=0 tx=0).stop 2024-04-25T01:22:32.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.633+0000 7f7a0a612640 1 -- 172.21.15.77:0/3302495774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a0415e1c0 msgr2=0x7f7a041f6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.633+0000 7f7a0a612640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a0415e1c0 0x7f7a041f6fd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f79f802a810 tx=0x7f79f8004730 comp rx=0 tx=0).stop 2024-04-25T01:22:32.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.633+0000 7f7a0a612640 1 -- 172.21.15.77:0/3302495774 shutdown_connections 2024-04-25T01:22:32.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.634+0000 7f7a0a612640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f79f006be60 0x7f79f006e320 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.634+0000 7f7a0a612640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a0415e1c0 0x7f7a041f6fd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.634+0000 7f7a0a612640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a041f4230 0x7f7a040ce1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.634+0000 7f7a0a612640 1 --2- 172.21.15.77:0/3302495774 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79ec003f30 0x7f7a04141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.634+0000 7f7a0a612640 1 -- 172.21.15.77:0/3302495774 >> 172.21.15.77:0/3302495774 conn(0x7f7a0406b720 msgr2=0x7f7a04071560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:32.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.634+0000 7f7a0a612640 1 -- 172.21.15.77:0/3302495774 shutdown_connections 2024-04-25T01:22:32.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.634+0000 7f7a0a612640 1 -- 172.21.15.77:0/3302495774 wait complete. 2024-04-25T01:22:32.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.744+0000 7ff01a542640 1 Processor -- start 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.744+0000 7ff01a542640 1 -- start start 2024-04-25T01:22:32.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.745+0000 7ff01a542640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0140cd880 0x7ff01406b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.745+0000 7ff01a542640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0140b0200 con 0x7ff01406b660 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.745+0000 7ff01a542640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0140b5840 con 0x7ff0140af8c0 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.745+0000 7ff01a542640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff01406ba90 con 0x7ff0140cd880 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.745+0000 7ff018adf640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff01406b660 0x7ff01415d940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57468/0 (socket says 172.21.15.77:57468) 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.745+0000 7ff018adf640 1 -- 172.21.15.77:0/2463823234 learned_addr learned my addr 172.21.15.77:0/2463823234 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.745+0000 7ff0137fe640 1 --2- 172.21.15.77:0/2463823234 >> v2:172.21.15.121:3300/0 conn(0x7ff0140cd880 0x7ff01406b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.745+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2760226539 0 0) 0x7ff0140b0200 con 0x7ff01406b660 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feffc003520 con 0x7ff01406b660 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1087423635 0 0) 0x7feffc003520 con 0x7ff01406b660 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 >> v2:172.21.15.121:3300/0 conn(0x7ff0140cd880 msgr2=0x7ff01406b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 --2- 172.21.15.77:0/2463823234 >> v2:172.21.15.121:3300/0 conn(0x7ff0140cd880 0x7ff01406b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 >> v1:172.21.15.121:6789/0 conn(0x7ff0140af8c0 legacy=0x7ff0140b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0140b5b20 con 0x7ff01406b660 2024-04-25T01:22:32.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7ff00c0019f0 con 0x7ff01406b660 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 >> v1:172.21.15.77:6789/0 conn(0x7ff01406b660 legacy=0x7ff01415d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 --2- 172.21.15.77:0/2463823234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feffc0039b0 0x7feffc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 --2- 172.21.15.77:0/2463823234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc004800 0x7feffc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 --2- 172.21.15.77:0/2463823234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feffc0051c0 0x7feffc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff01406ba90 con 0x7feffc004800 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feffc00c160 con 0x7feffc0039b0 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7feffc00c360 con 0x7feffc0051c0 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff0137fe640 1 --2- 172.21.15.77:0/2463823234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc004800 0x7feffc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff013fff640 1 --2- 172.21.15.77:0/2463823234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feffc0039b0 0x7feffc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff018adf640 1 --2- 172.21.15.77:0/2463823234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feffc0051c0 0x7feffc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff0137fe640 1 -- 172.21.15.77:0/2463823234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feffc0051c0 msgr2=0x7feffc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff0137fe640 1 --2- 172.21.15.77:0/2463823234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feffc0051c0 0x7feffc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff0137fe640 1 -- 172.21.15.77:0/2463823234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feffc0039b0 msgr2=0x7feffc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:32.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff0137fe640 1 --2- 172.21.15.77:0/2463823234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feffc0039b0 0x7feffc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff0137fe640 1 -- 172.21.15.77:0/2463823234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc003520 con 0x7feffc004800 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.746+0000 7ff018adf640 1 -- 172.21.15.77:0/2463823234 reap_dead start 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.747+0000 7ff0137fe640 1 --2- 172.21.15.77:0/2463823234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc004800 0x7feffc004c80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff008009ac0 tx=0x7feffc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a48bf635326044 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.747+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff008036070 con 0x7feffc004800 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.747+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff008031400 con 0x7feffc004800 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.747+0000 7ff012ffd640 1 -- 172.21.15.77:0/2463823234 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff008009690 con 0x7feffc004800 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.747+0000 7ff01a542640 1 -- 172.21.15.77:0/2463823234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc004800 msgr2=0x7feffc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.747+0000 7ff01a542640 1 --2- 172.21.15.77:0/2463823234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc004800 0x7feffc004c80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff008009ac0 tx=0x7feffc005830 comp rx=0 tx=0).stop 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.747+0000 7ff01a542640 1 -- 172.21.15.77:0/2463823234 shutdown_connections 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.747+0000 7ff01a542640 1 --2- 172.21.15.77:0/2463823234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc004800 0x7feffc004c80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.747+0000 7ff01a542640 1 -- 172.21.15.77:0/2463823234 >> 172.21.15.77:0/2463823234 conn(0x7ff0140b1f00 msgr2=0x7ff0140b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.748+0000 7ff01a542640 1 -- 172.21.15.77:0/2463823234 shutdown_connections 2024-04-25T01:22:32.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.748+0000 7ff01a542640 1 -- 172.21.15.77:0/2463823234 wait complete. 2024-04-25T01:22:32.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.749+0000 7ff01a542640 1 Processor -- start 2024-04-25T01:22:32.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.749+0000 7ff01a542640 1 -- start start 2024-04-25T01:22:32.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff01a542640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feffc004800 0x7ff014155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff01a542640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff01415c900 0x7ff0140cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff01a542640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0140c9790 0x7ff0140c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff01a542640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff0140afca0 con 0x7ff01415c900 2024-04-25T01:22:32.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff01a542640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff0140b60e0 con 0x7feffc004800 2024-04-25T01:22:32.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff01a542640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff0140afe20 con 0x7ff0140c9790 2024-04-25T01:22:32.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff0137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff01415c900 0x7ff0140cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff0137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff01415c900 0x7ff0140cf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49200/0 (socket says 172.21.15.77:49200) 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff0137fe640 1 -- 172.21.15.77:0/2152048182 learned_addr learned my addr 172.21.15.77:0/2152048182 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff013fff640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feffc004800 0x7ff014155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff0137fe640 1 -- 172.21.15.77:0/2152048182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0140c9790 msgr2=0x7ff0140c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff0137fe640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0140c9790 0x7ff0140c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff0137fe640 1 -- 172.21.15.77:0/2152048182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feffc004800 msgr2=0x7ff014155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff0137fe640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feffc004800 0x7ff014155cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff0137fe640 1 -- 172.21.15.77:0/2152048182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0140cfe80 con 0x7ff01415c900 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff0137fe640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff01415c900 0x7ff0140cf940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff00802abb0 tx=0x7ff008004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff010ff9640 1 -- 172.21.15.77:0/2152048182 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00803f020 con 0x7ff01415c900 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff010ff9640 1 -- 172.21.15.77:0/2152048182 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff008037040 con 0x7ff01415c900 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff01a542640 1 -- 172.21.15.77:0/2152048182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff01415eca0 con 0x7ff01415c900 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff010ff9640 1 -- 172.21.15.77:0/2152048182 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff008036070 con 0x7ff01415c900 2024-04-25T01:22:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.750+0000 7ff01a542640 1 -- 172.21.15.77:0/2152048182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff01415f260 con 0x7ff01415c900 2024-04-25T01:22:32.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.752+0000 7ff01a542640 1 -- 172.21.15.77:0/2152048182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff014068fb0 con 0x7ff01415c900 2024-04-25T01:22:32.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.752+0000 7ff010ff9640 1 -- 172.21.15.77:0/2152048182 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff008042020 con 0x7ff01415c900 2024-04-25T01:22:32.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.752+0000 7ff010ff9640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff00006be60 0x7ff00006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.752+0000 7ff013fff640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff00006be60 0x7ff00006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.752+0000 7ff010ff9640 1 -- 172.21.15.77:0/2152048182 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7ff0080ab250 con 0x7ff01415c900 2024-04-25T01:22:32.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.753+0000 7ff013fff640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff00006be60 0x7ff00006e320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff004000c00 tx=0x7ff004009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.756+0000 7ff010ff9640 1 -- 172.21.15.77:0/2152048182 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff008049030 con 0x7ff01415c900 2024-04-25T01:22:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.927+0000 7ff01a542640 1 -- 172.21.15.77:0/2152048182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff01406edc0 con 0x7ff01415c900 2024-04-25T01:22:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.928+0000 7ff010ff9640 1 -- 172.21.15.77:0/2152048182 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7ff0140cfe80 con 0x7ff01415c900 2024-04-25T01:22:32.910 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:32.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 -- 172.21.15.77:0/2152048182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff00006be60 msgr2=0x7ff00006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff00006be60 0x7ff00006e320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff004000c00 tx=0x7ff004009040 comp rx=0 tx=0).stop 2024-04-25T01:22:32.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 -- 172.21.15.77:0/2152048182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff01415c900 msgr2=0x7ff0140cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff01415c900 0x7ff0140cf940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff00802abb0 tx=0x7ff008004730 comp rx=0 tx=0).stop 2024-04-25T01:22:32.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 -- 172.21.15.77:0/2152048182 shutdown_connections 2024-04-25T01:22:32.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff00006be60 0x7ff00006e320 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff01415c900 0x7ff0140cf940 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0140c9790 0x7ff0140c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 --2- 172.21.15.77:0/2152048182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feffc004800 0x7ff014155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 -- 172.21.15.77:0/2152048182 >> 172.21.15.77:0/2152048182 conn(0x7ff0140b1f00 msgr2=0x7ff0140cbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:32.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 -- 172.21.15.77:0/2152048182 shutdown_connections 2024-04-25T01:22:32.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:32.930+0000 7ff01a542640 1 -- 172.21.15.77:0/2152048182 wait complete. 2024-04-25T01:22:32.922 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:33.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.040+0000 7fa07468c640 1 Processor -- start 2024-04-25T01:22:33.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.040+0000 7fa07468c640 1 -- start start 2024-04-25T01:22:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.041+0000 7fa07468c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa06c140d30 0x7fa06c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.041+0000 7fa07468c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06c0ca790 con 0x7fa06c162360 2024-04-25T01:22:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.041+0000 7fa07468c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06c0caa70 con 0x7fa06c06b120 2024-04-25T01:22:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.041+0000 7fa07468c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fa06c0cad50 con 0x7fa06c140d30 2024-04-25T01:22:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.041+0000 7fa072428640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa06c140d30 0x7fa06c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.041+0000 7fa071c27640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa06c162360 0x7fa06c0c9f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57480/0 (socket says 172.21.15.77:57480) 2024-04-25T01:22:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.041+0000 7fa071c27640 1 -- 172.21.15.77:0/2193353944 learned_addr learned my addr 172.21.15.77:0/2193353944 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.042+0000 7fa072428640 1 -- 172.21.15.77:0/2193353944 >> v1:172.21.15.121:6789/0 conn(0x7fa06c06b120 legacy=0x7fa06c143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.042+0000 7fa072428640 1 -- 172.21.15.77:0/2193353944 >> v1:172.21.15.77:6789/0 conn(0x7fa06c162360 legacy=0x7fa06c0c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.042+0000 7fa071426640 1 -- 172.21.15.77:0/2193353944 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4261843516 0 0) 0x7fa06c0ca790 con 0x7fa06c162360 2024-04-25T01:22:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.042+0000 7fa072428640 1 -- 172.21.15.77:0/2193353944 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06c161b60 con 0x7fa06c140d30 2024-04-25T01:22:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.042+0000 7fa072428640 1 --2- 172.21.15.77:0/2193353944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa06c140d30 0x7fa06c15f050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa068009d90 tx=0x7fa06802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d027f46c9db95f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.042+0000 7fa071426640 1 -- 172.21.15.77:0/2193353944 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068007740 con 0x7fa06c140d30 2024-04-25T01:22:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.042+0000 7fa071426640 1 -- 172.21.15.77:0/2193353944 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0680078e0 con 0x7fa06c140d30 2024-04-25T01:22:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.042+0000 7fa071426640 1 -- 172.21.15.77:0/2193353944 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068007cb0 con 0x7fa06c140d30 2024-04-25T01:22:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.043+0000 7fa07468c640 1 -- 172.21.15.77:0/2193353944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa06c140d30 msgr2=0x7fa06c15f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.043+0000 7fa07468c640 1 --2- 172.21.15.77:0/2193353944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa06c140d30 0x7fa06c15f050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa068009d90 tx=0x7fa06802a6a0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.043+0000 7fa07468c640 1 -- 172.21.15.77:0/2193353944 shutdown_connections 2024-04-25T01:22:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.043+0000 7fa07468c640 1 --2- 172.21.15.77:0/2193353944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa06c140d30 0x7fa06c15f050 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.043+0000 7fa07468c640 1 -- 172.21.15.77:0/2193353944 >> 172.21.15.77:0/2193353944 conn(0x7fa06c0d0bf0 msgr2=0x7fa06c0d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.043+0000 7fa07468c640 1 -- 172.21.15.77:0/2193353944 shutdown_connections 2024-04-25T01:22:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.043+0000 7fa07468c640 1 -- 172.21.15.77:0/2193353944 wait complete. 2024-04-25T01:22:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.044+0000 7fa07468c640 1 Processor -- start 2024-04-25T01:22:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.044+0000 7fa07468c640 1 -- start start 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa07468c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa06c06b120 0x7fa06c1f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa07468c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa06c140d30 0x7fa06c140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa07468c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa06c162360 0x7fa06c158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa07468c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c162940 con 0x7fa06c162360 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa07468c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c1627c0 con 0x7fa06c140d30 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa07468c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa06c141110 con 0x7fa06c06b120 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa072c29640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa06c162360 0x7fa06c158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa072c29640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa06c162360 0x7fa06c158b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49222/0 (socket says 172.21.15.77:49222) 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa072c29640 1 -- 172.21.15.77:0/70128711 learned_addr learned my addr 172.21.15.77:0/70128711 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa071c27640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa06c140d30 0x7fa06c140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa072428640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa06c06b120 0x7fa06c1f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa072c29640 1 -- 172.21.15.77:0/70128711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa06c06b120 msgr2=0x7fa06c1f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa072c29640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa06c06b120 0x7fa06c1f4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa072c29640 1 -- 172.21.15.77:0/70128711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa06c140d30 msgr2=0x7fa06c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa072c29640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa06c140d30 0x7fa06c140660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa072c29640 1 -- 172.21.15.77:0/70128711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06c1590d0 con 0x7fa06c162360 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa071c27640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa06c140d30 0x7fa06c140660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa072c29640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa06c162360 0x7fa06c158b90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa06400b490 tx=0x7fa06400b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.045+0000 7fa062ffd640 1 -- 172.21.15.77:0/70128711 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa064015020 con 0x7fa06c162360 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.046+0000 7fa062ffd640 1 -- 172.21.15.77:0/70128711 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa064011850 con 0x7fa06c162360 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.046+0000 7fa07468c640 1 -- 172.21.15.77:0/70128711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa06c1593a0 con 0x7fa06c162360 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.046+0000 7fa062ffd640 1 -- 172.21.15.77:0/70128711 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa064011c40 con 0x7fa06c162360 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.046+0000 7fa07468c640 1 -- 172.21.15.77:0/70128711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa06c159960 con 0x7fa06c162360 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.047+0000 7fa07468c640 1 -- 172.21.15.77:0/70128711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa06c0cf5f0 con 0x7fa06c162360 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.047+0000 7fa062ffd640 1 -- 172.21.15.77:0/70128711 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa064018020 con 0x7fa06c162360 2024-04-25T01:22:33.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.047+0000 7fa062ffd640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa05006be30 0x7fa05006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:33.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.047+0000 7fa062ffd640 1 -- 172.21.15.77:0/70128711 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fa06408b040 con 0x7fa06c162360 2024-04-25T01:22:33.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.047+0000 7fa072428640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa05006be30 0x7fa05006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:33.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.048+0000 7fa072428640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa05006be30 0x7fa05006e2f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa068034000 tx=0x7fa06802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:33.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.052+0000 7fa062ffd640 1 -- 172.21.15.77:0/70128711 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa06c0cf5f0 con 0x7fa06c162360 2024-04-25T01:22:33.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.228+0000 7fa07468c640 1 -- 172.21.15.77:0/70128711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa06c068fb0 con 0x7fa06c162360 2024-04-25T01:22:33.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.228+0000 7fa062ffd640 1 -- 172.21.15.77:0/70128711 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fa06401f020 con 0x7fa06c162360 2024-04-25T01:22:33.210 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:33.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.230+0000 7fa07468c640 1 -- 172.21.15.77:0/70128711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa05006be30 msgr2=0x7fa05006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:33.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.230+0000 7fa07468c640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa05006be30 0x7fa05006e2f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa068034000 tx=0x7fa06802f000 comp rx=0 tx=0).stop 2024-04-25T01:22:33.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.230+0000 7fa07468c640 1 -- 172.21.15.77:0/70128711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa06c162360 msgr2=0x7fa06c158b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:33.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.230+0000 7fa07468c640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa06c162360 0x7fa06c158b90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa06400b490 tx=0x7fa06400b950 comp rx=0 tx=0).stop 2024-04-25T01:22:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.230+0000 7fa07468c640 1 -- 172.21.15.77:0/70128711 shutdown_connections 2024-04-25T01:22:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.230+0000 7fa07468c640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa05006be30 0x7fa05006e2f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.230+0000 7fa07468c640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa06c162360 0x7fa06c158b90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.230+0000 7fa07468c640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa06c140d30 0x7fa06c140660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.230+0000 7fa07468c640 1 --2- 172.21.15.77:0/70128711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa06c06b120 0x7fa06c1f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.230+0000 7fa07468c640 1 -- 172.21.15.77:0/70128711 >> 172.21.15.77:0/70128711 conn(0x7fa06c0d0bf0 msgr2=0x7fa06c0b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.231+0000 7fa07468c640 1 -- 172.21.15.77:0/70128711 shutdown_connections 2024-04-25T01:22:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.231+0000 7fa07468c640 1 -- 172.21.15.77:0/70128711 wait complete. 2024-04-25T01:22:33.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_backtrace_repair' 2024-04-25T01:22:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.339+0000 7fedc436a640 1 Processor -- start 2024-04-25T01:22:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.339+0000 7fedc436a640 1 -- start start 2024-04-25T01:22:33.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc436a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fedbc154ac0 0x7fedbc15b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:33.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc436a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedbc0cf2a0 0x7fedbc0711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:33.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc436a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedbc154ea0 con 0x7fedbc071730 2024-04-25T01:22:33.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc436a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fedbc06d540 con 0x7fedbc154ac0 2024-04-25T01:22:33.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc436a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fedbc06d790 con 0x7fedbc0cf2a0 2024-04-25T01:22:33.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc2106640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fedbc154ac0 0x7fedbc15b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc2106640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fedbc154ac0 0x7fedbc15b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49226/0 (socket says 172.21.15.77:49226) 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc2106640 1 -- 172.21.15.77:0/4187662941 learned_addr learned my addr 172.21.15.77:0/4187662941 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc1905640 1 --2- 172.21.15.77:0/4187662941 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedbc0cf2a0 0x7fedbc0711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc2106640 1 -- 172.21.15.77:0/4187662941 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedbc0cf2a0 msgr2=0x7fedbc0711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc2106640 1 --2- 172.21.15.77:0/4187662941 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedbc0cf2a0 0x7fedbc0711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc2106640 1 -- 172.21.15.77:0/4187662941 >> v1:172.21.15.121:6789/0 conn(0x7fedbc071730 legacy=0x7fedbc06e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.340+0000 7fedc2106640 1 -- 172.21.15.77:0/4187662941 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedbc06da70 con 0x7fedbc154ac0 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.341+0000 7fedc2106640 1 --2- 172.21.15.77:0/4187662941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedbc154ac0 0x7fedbc15b5f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009870 tx=0x7fedb802b320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3571eeacc03a405 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.341+0000 7fedc1104640 1 -- 172.21.15.77:0/4187662941 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8038070 con 0x7fedbc154ac0 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.341+0000 7fedc1104640 1 -- 172.21.15.77:0/4187662941 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedb8033640 con 0x7fedbc154ac0 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.341+0000 7fedc1104640 1 -- 172.21.15.77:0/4187662941 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8033a10 con 0x7fedbc154ac0 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.341+0000 7fedc436a640 1 -- 172.21.15.77:0/4187662941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedbc154ac0 msgr2=0x7fedbc15b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.341+0000 7fedc436a640 1 --2- 172.21.15.77:0/4187662941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedbc154ac0 0x7fedbc15b5f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009870 tx=0x7fedb802b320 comp rx=0 tx=0).stop 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.342+0000 7fedc436a640 1 -- 172.21.15.77:0/4187662941 shutdown_connections 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.342+0000 7fedc436a640 1 --2- 172.21.15.77:0/4187662941 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedbc0cf2a0 0x7fedbc0711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.342+0000 7fedc436a640 1 --2- 172.21.15.77:0/4187662941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedbc154ac0 0x7fedbc15b5f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.342+0000 7fedc436a640 1 -- 172.21.15.77:0/4187662941 >> 172.21.15.77:0/4187662941 conn(0x7fedbc0b1820 msgr2=0x7fedbc0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:33.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.342+0000 7fedc436a640 1 -- 172.21.15.77:0/4187662941 shutdown_connections 2024-04-25T01:22:33.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.342+0000 7fedc436a640 1 -- 172.21.15.77:0/4187662941 wait complete. 2024-04-25T01:22:33.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.343+0000 7fedc436a640 1 Processor -- start 2024-04-25T01:22:33.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.343+0000 7fedc436a640 1 -- start start 2024-04-25T01:22:33.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.343+0000 7fedc436a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedbc071730 0x7fedbc150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:33.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.343+0000 7fedc436a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedbc0cf2a0 0x7fedbc14fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:33.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.343+0000 7fedc436a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedbc154ac0 0x7fedbc15d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:33.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc436a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc0cf720 con 0x7fedbc154ac0 2024-04-25T01:22:33.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc436a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc06e030 con 0x7fedbc0cf2a0 2024-04-25T01:22:33.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc436a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fedbc06ec60 con 0x7fedbc071730 2024-04-25T01:22:33.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc2907640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedbc154ac0 0x7fedbc15d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc2907640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedbc154ac0 0x7fedbc15d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49228/0 (socket says 172.21.15.77:49228) 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc2907640 1 -- 172.21.15.77:0/1050695391 learned_addr learned my addr 172.21.15.77:0/1050695391 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc2106640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedbc071730 0x7fedbc150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc1905640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedbc0cf2a0 0x7fedbc14fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc2907640 1 -- 172.21.15.77:0/1050695391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedbc071730 msgr2=0x7fedbc150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc2907640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedbc071730 0x7fedbc150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc2907640 1 -- 172.21.15.77:0/1050695391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedbc0cf2a0 msgr2=0x7fedbc14fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc2907640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedbc0cf2a0 0x7fedbc14fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc2907640 1 -- 172.21.15.77:0/1050695391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedbc155110 con 0x7fedbc154ac0 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc2907640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedbc154ac0 0x7fedbc15d490 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fedb400c7b0 tx=0x7fedb400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedb2ffd640 1 -- 172.21.15.77:0/1050695391 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb4004b60 con 0x7fedbc154ac0 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedb2ffd640 1 -- 172.21.15.77:0/1050695391 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedb4004d00 con 0x7fedbc154ac0 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedb2ffd640 1 -- 172.21.15.77:0/1050695391 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb400aa90 con 0x7fedbc154ac0 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc436a640 1 -- 172.21.15.77:0/1050695391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedbc1553f0 con 0x7fedbc154ac0 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.344+0000 7fedc436a640 1 -- 172.21.15.77:0/1050695391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedbc1559b0 con 0x7fedbc154ac0 2024-04-25T01:22:33.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.345+0000 7fedc436a640 1 -- 172.21.15.77:0/1050695391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedbc068fb0 con 0x7fedbc154ac0 2024-04-25T01:22:33.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.346+0000 7fedb2ffd640 1 -- 172.21.15.77:0/1050695391 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fedb400ac30 con 0x7fedbc154ac0 2024-04-25T01:22:33.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.346+0000 7fedb2ffd640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feda006be30 0x7feda006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:33.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.346+0000 7fedb2ffd640 1 -- 172.21.15.77:0/1050695391 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fedb408b610 con 0x7fedbc154ac0 2024-04-25T01:22:33.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.346+0000 7fedc2106640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feda006be30 0x7feda006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:33.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.346+0000 7fedc2106640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feda006be30 0x7feda006e2f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fedb8035000 tx=0x7fedb802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:33.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.350+0000 7fedb2ffd640 1 -- 172.21.15.77:0/1050695391 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fedb4020070 con 0x7fedbc154ac0 2024-04-25T01:22:33.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:33.591+0000 7fedc436a640 1 -- 172.21.15.77:0/1050695391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_backtrace_repair"]} v 0) -- 0x7fedbc070fd0 con 0x7fedbc154ac0 2024-04-25T01:22:34.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.323+0000 7fedb2ffd640 1 -- 172.21.15.77:0/1050695391 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_backtrace_repair"]}]=0 v69) ==== 151+0+0 (secure 0 0 0) 0x7fedb4059b30 con 0x7fedbc154ac0 2024-04-25T01:22:34.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 -- 172.21.15.77:0/1050695391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feda006be30 msgr2=0x7feda006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feda006be30 0x7feda006e2f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fedb8035000 tx=0x7fedb802f000 comp rx=0 tx=0).stop 2024-04-25T01:22:34.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 -- 172.21.15.77:0/1050695391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedbc154ac0 msgr2=0x7fedbc15d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedbc154ac0 0x7fedbc15d490 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fedb400c7b0 tx=0x7fedb400cc70 comp rx=0 tx=0).stop 2024-04-25T01:22:34.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 -- 172.21.15.77:0/1050695391 shutdown_connections 2024-04-25T01:22:34.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feda006be30 0x7feda006e2f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedbc154ac0 0x7fedbc15d490 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedbc0cf2a0 0x7fedbc14fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 --2- 172.21.15.77:0/1050695391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedbc071730 0x7fedbc150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 -- 172.21.15.77:0/1050695391 >> 172.21.15.77:0/1050695391 conn(0x7fedbc0b1820 msgr2=0x7fedbc0b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:34.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 -- 172.21.15.77:0/1050695391 shutdown_connections 2024-04-25T01:22:34.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.326+0000 7fedc436a640 1 -- 172.21.15.77:0/1050695391 wait complete. 2024-04-25T01:22:34.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T01:22:34.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.445+0000 7f34dc888640 1 Processor -- start 2024-04-25T01:22:34.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.445+0000 7f34dc888640 1 -- start start 2024-04-25T01:22:34.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34dc888640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f34d806caf0 0x7f34d80b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34dc888640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34d806da80 0x7f34d8070b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34dc888640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34d815bb80 con 0x7f34d80720c0 2024-04-25T01:22:34.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34dc888640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f34d80724a0 con 0x7f34d806da80 2024-04-25T01:22:34.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34dc888640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f34d80b3750 con 0x7f34d806caf0 2024-04-25T01:22:34.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34d6d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34d806da80 0x7f34d8070b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34d659c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f34d80720c0 0x7f34d806d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57484/0 (socket says 172.21.15.77:57484) 2024-04-25T01:22:34.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34d659c640 1 -- 172.21.15.77:0/980945310 learned_addr learned my addr 172.21.15.77:0/980945310 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:34.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34d6d9d640 1 -- 172.21.15.77:0/980945310 >> v2:172.21.15.121:3300/0 conn(0x7f34d806caf0 msgr2=0x7f34d80b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:34.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34d5d9b640 1 --2- 172.21.15.77:0/980945310 >> v2:172.21.15.121:3300/0 conn(0x7f34d806caf0 0x7f34d80b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34d559a640 1 -- 172.21.15.77:0/980945310 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1515931680 0 0) 0x7f34d815bb80 con 0x7f34d80720c0 2024-04-25T01:22:34.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34d6d9d640 1 --2- 172.21.15.77:0/980945310 >> v2:172.21.15.121:3300/0 conn(0x7f34d806caf0 0x7f34d80b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34d6d9d640 1 -- 172.21.15.77:0/980945310 >> v1:172.21.15.77:6789/0 conn(0x7f34d80720c0 legacy=0x7f34d806d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34d6d9d640 1 -- 172.21.15.77:0/980945310 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d80b3a30 con 0x7f34d806da80 2024-04-25T01:22:34.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.446+0000 7f34d5d9b640 1 --2- 172.21.15.77:0/980945310 >> v2:172.21.15.121:3300/0 conn(0x7f34d806caf0 0x7f34d80b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.447+0000 7f34d6d9d640 1 --2- 172.21.15.77:0/980945310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d806da80 0x7f34d8070b50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f34d806ef50 tx=0x7f34d002b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd1478ea2204e4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.447+0000 7f34d559a640 1 -- 172.21.15.77:0/980945310 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d000b040 con 0x7f34d806da80 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.447+0000 7f34d559a640 1 -- 172.21.15.77:0/980945310 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d0030710 con 0x7f34d806da80 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.447+0000 7f34d559a640 1 -- 172.21.15.77:0/980945310 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0030ae0 con 0x7f34d806da80 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.448+0000 7f34dc888640 1 -- 172.21.15.77:0/980945310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d806da80 msgr2=0x7f34d8070b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.448+0000 7f34dc888640 1 --2- 172.21.15.77:0/980945310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d806da80 0x7f34d8070b50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f34d806ef50 tx=0x7f34d002b4d0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.448+0000 7f34dc888640 1 -- 172.21.15.77:0/980945310 shutdown_connections 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.448+0000 7f34dc888640 1 --2- 172.21.15.77:0/980945310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d806da80 0x7f34d8070b50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.448+0000 7f34dc888640 1 --2- 172.21.15.77:0/980945310 >> v2:172.21.15.121:3300/0 conn(0x7f34d806caf0 0x7f34d80b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.448+0000 7f34dc888640 1 -- 172.21.15.77:0/980945310 >> 172.21.15.77:0/980945310 conn(0x7f34d806b720 msgr2=0x7f34d80b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.448+0000 7f34dc888640 1 -- 172.21.15.77:0/980945310 shutdown_connections 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.448+0000 7f34dc888640 1 -- 172.21.15.77:0/980945310 wait complete. 2024-04-25T01:22:34.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.449+0000 7f34dc888640 1 Processor -- start 2024-04-25T01:22:34.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.449+0000 7f34dc888640 1 -- start start 2024-04-25T01:22:34.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34dc888640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d806caf0 0x7f34d81f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34dc888640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d806da80 0x7f34d81413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34dc888640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34d80720c0 0x7f34d80b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34dc888640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f34d815b6a0 con 0x7f34d806caf0 2024-04-25T01:22:34.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34dc888640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f34d80b3ff0 con 0x7f34d806da80 2024-04-25T01:22:34.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34dc888640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f34d815b820 con 0x7f34d80720c0 2024-04-25T01:22:34.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34d659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d806caf0 0x7f34d81f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34d659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d806caf0 0x7f34d81f1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49244/0 (socket says 172.21.15.77:49244) 2024-04-25T01:22:34.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34d659c640 1 -- 172.21.15.77:0/1565557447 learned_addr learned my addr 172.21.15.77:0/1565557447 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:34.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34d659c640 1 -- 172.21.15.77:0/1565557447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34d80720c0 msgr2=0x7f34d80b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34d659c640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34d80720c0 0x7f34d80b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34d659c640 1 -- 172.21.15.77:0/1565557447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d806da80 msgr2=0x7f34d81413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34d5d9b640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d806da80 0x7f34d81413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34d659c640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d806da80 0x7f34d81413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34d659c640 1 -- 172.21.15.77:0/1565557447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d81f30a0 con 0x7f34d806caf0 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.450+0000 7f34d5d9b640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d806da80 0x7f34d81413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.451+0000 7f34d659c640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d806caf0 0x7f34d81f1ec0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f34c800c7c0 tx=0x7f34c800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.451+0000 7f34c6ffd640 1 -- 172.21.15.77:0/1565557447 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8018070 con 0x7f34d806caf0 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.451+0000 7f34c6ffd640 1 -- 172.21.15.77:0/1565557447 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c800a5e0 con 0x7f34d806caf0 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.451+0000 7f34c6ffd640 1 -- 172.21.15.77:0/1565557447 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c800a9d0 con 0x7f34d806caf0 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.451+0000 7f34dc888640 1 -- 172.21.15.77:0/1565557447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d81f2740 con 0x7f34d806caf0 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.451+0000 7f34dc888640 1 -- 172.21.15.77:0/1565557447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d81f2d00 con 0x7f34d806caf0 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.452+0000 7f34dc888640 1 -- 172.21.15.77:0/1565557447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d8068fb0 con 0x7f34d806caf0 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.452+0000 7f34c6ffd640 1 -- 172.21.15.77:0/1565557447 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f34c800ab70 con 0x7f34d806caf0 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.453+0000 7f34c6ffd640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34b806be30 0x7f34b806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.453+0000 7f34c6ffd640 1 -- 172.21.15.77:0/1565557447 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f34c808a8e0 con 0x7f34d806caf0 2024-04-25T01:22:34.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.453+0000 7f34d5d9b640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34b806be30 0x7f34b806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.453+0000 7f34d5d9b640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34b806be30 0x7f34b806e2f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f34cc006fd0 tx=0x7f34cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:34.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.457+0000 7f34c6ffd640 1 -- 172.21.15.77:0/1565557447 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f34d81f30a0 con 0x7f34d806caf0 2024-04-25T01:22:34.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.632+0000 7f34dc888640 1 -- 172.21.15.77:0/1565557447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f34d80b1c70 con 0x7f34d806caf0 2024-04-25T01:22:34.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.635+0000 7f34c6ffd640 1 -- 172.21.15.77:0/1565557447 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f34c8088e20 con 0x7f34d806caf0 2024-04-25T01:22:34.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 -- 172.21.15.77:0/1565557447 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34b806be30 msgr2=0x7f34b806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34b806be30 0x7f34b806e2f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f34cc006fd0 tx=0x7f34cc008040 comp rx=0 tx=0).stop 2024-04-25T01:22:34.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 -- 172.21.15.77:0/1565557447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d806caf0 msgr2=0x7f34d81f1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d806caf0 0x7f34d81f1ec0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f34c800c7c0 tx=0x7f34c800cc80 comp rx=0 tx=0).stop 2024-04-25T01:22:34.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 -- 172.21.15.77:0/1565557447 shutdown_connections 2024-04-25T01:22:34.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34b806be30 0x7f34b806e2f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34d80720c0 0x7f34d80b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d806da80 0x7f34d81413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 --2- 172.21.15.77:0/1565557447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d806caf0 0x7f34d81f1ec0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 -- 172.21.15.77:0/1565557447 >> 172.21.15.77:0/1565557447 conn(0x7f34d806b720 msgr2=0x7f34d80af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:34.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 -- 172.21.15.77:0/1565557447 shutdown_connections 2024-04-25T01:22:34.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.637+0000 7f34dc888640 1 -- 172.21.15.77:0/1565557447 wait complete. 2024-04-25T01:22:34.630 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T01:22:34.630 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:22:34.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:22:34.717 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T01:22:34.717 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:22:34.717 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:34.732 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:22:34.733 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:34.788 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:22:34.788 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:34.844 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:34.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.998+0000 7f631f6bd640 1 Processor -- start 2024-04-25T01:22:34.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.998+0000 7f631f6bd640 1 -- start start 2024-04-25T01:22:34.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.999+0000 7f631f6bd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6318143940 0x7f631806e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.999+0000 7f631f6bd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f631815f400 0x7f631806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.999+0000 7f631f6bd640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6318070e60 con 0x7f6318143d20 2024-04-25T01:22:34.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.999+0000 7f631f6bd640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6318071140 con 0x7f631815f400 2024-04-25T01:22:34.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.999+0000 7f631f6bd640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6318071420 con 0x7f6318143940 2024-04-25T01:22:34.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.999+0000 7f631dc5a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f631815f400 0x7f631806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.999+0000 7f631cc58640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6318143d20 0x7f631815ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57496/0 (socket says 172.21.15.77:57496) 2024-04-25T01:22:34.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.999+0000 7f631cc58640 1 -- 172.21.15.77:0/3868324959 learned_addr learned my addr 172.21.15.77:0/3868324959 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:34.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:34.999+0000 7f631d459640 1 --2- 172.21.15.77:0/3868324959 >> v2:172.21.15.121:3300/0 conn(0x7f6318143940 0x7f631806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.000+0000 7f631dc5a640 1 -- 172.21.15.77:0/3868324959 >> v2:172.21.15.121:3300/0 conn(0x7f6318143940 msgr2=0x7f631806e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:34.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.000+0000 7f631dc5a640 1 --2- 172.21.15.77:0/3868324959 >> v2:172.21.15.121:3300/0 conn(0x7f6318143940 0x7f631806e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.000+0000 7f631dc5a640 1 -- 172.21.15.77:0/3868324959 >> v1:172.21.15.77:6789/0 conn(0x7f6318143d20 legacy=0x7f631815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.000+0000 7f631dc5a640 1 -- 172.21.15.77:0/3868324959 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6318071700 con 0x7f631815f400 2024-04-25T01:22:34.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.000+0000 7f630ffff640 1 -- 172.21.15.77:0/3868324959 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1544366123 0 0) 0x7f6318070e60 con 0x7f6318143d20 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.000+0000 7f631dc5a640 1 --2- 172.21.15.77:0/3868324959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f631815f400 0x7f631806eb90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f63100097a0 tx=0x7f631002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c931e848324d5bad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.000+0000 7f630ffff640 1 -- 172.21.15.77:0/3868324959 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310037070 con 0x7f631815f400 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.000+0000 7f630ffff640 1 -- 172.21.15.77:0/3868324959 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6310032440 con 0x7f631815f400 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.000+0000 7f630ffff640 1 -- 172.21.15.77:0/3868324959 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310032810 con 0x7f631815f400 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.001+0000 7f631f6bd640 1 -- 172.21.15.77:0/3868324959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f631815f400 msgr2=0x7f631806eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.001+0000 7f631f6bd640 1 --2- 172.21.15.77:0/3868324959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f631815f400 0x7f631806eb90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f63100097a0 tx=0x7f631002a1e0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.001+0000 7f631f6bd640 1 -- 172.21.15.77:0/3868324959 shutdown_connections 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.001+0000 7f631f6bd640 1 --2- 172.21.15.77:0/3868324959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f631815f400 0x7f631806eb90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.001+0000 7f631f6bd640 1 --2- 172.21.15.77:0/3868324959 >> v2:172.21.15.121:3300/0 conn(0x7f6318143940 0x7f631806e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.001+0000 7f631f6bd640 1 -- 172.21.15.77:0/3868324959 >> 172.21.15.77:0/3868324959 conn(0x7f63180b1820 msgr2=0x7f63180b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.001+0000 7f631f6bd640 1 -- 172.21.15.77:0/3868324959 shutdown_connections 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.001+0000 7f631f6bd640 1 -- 172.21.15.77:0/3868324959 wait complete. 2024-04-25T01:22:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.002+0000 7f631f6bd640 1 Processor -- start 2024-04-25T01:22:34.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.002+0000 7f631f6bd640 1 -- start start 2024-04-25T01:22:34.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631f6bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6318143940 0x7f6318156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631f6bd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6318143d20 0x7f6318156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631f6bd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f631815f400 0x7f631806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631f6bd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6318071ff0 con 0x7f6318143940 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631f6bd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6318071e70 con 0x7f631815f400 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631f6bd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6318072170 con 0x7f6318143d20 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631d459640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6318143940 0x7f6318156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631d459640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6318143940 0x7f6318156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49266/0 (socket says 172.21.15.77:49266) 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631d459640 1 -- 172.21.15.77:0/1055059564 learned_addr learned my addr 172.21.15.77:0/1055059564 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631d459640 1 -- 172.21.15.77:0/1055059564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6318143d20 msgr2=0x7f6318156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631d459640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6318143d20 0x7f6318156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631d459640 1 -- 172.21.15.77:0/1055059564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f631815f400 msgr2=0x7f631806d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631d459640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f631815f400 0x7f631806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631d459640 1 -- 172.21.15.77:0/1055059564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63181597f0 con 0x7f6318143940 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f631d459640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6318143940 0x7f6318156e90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6314007ef0 tx=0x7f631400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.003+0000 7f630dffb640 1 -- 172.21.15.77:0/1055059564 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6314014620 con 0x7f6318143940 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.004+0000 7f630dffb640 1 -- 172.21.15.77:0/1055059564 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6314019070 con 0x7f6318143940 2024-04-25T01:22:34.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.004+0000 7f631f6bd640 1 -- 172.21.15.77:0/1055059564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6318159ad0 con 0x7f6318143940 2024-04-25T01:22:34.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.004+0000 7f630dffb640 1 -- 172.21.15.77:0/1055059564 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6314014a00 con 0x7f6318143940 2024-04-25T01:22:34.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.004+0000 7f631f6bd640 1 -- 172.21.15.77:0/1055059564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63181573d0 con 0x7f6318143940 2024-04-25T01:22:34.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.005+0000 7f631f6bd640 1 -- 172.21.15.77:0/1055059564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6318068fb0 con 0x7f6318143940 2024-04-25T01:22:34.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.005+0000 7f630dffb640 1 -- 172.21.15.77:0/1055059564 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6314014ba0 con 0x7f6318143940 2024-04-25T01:22:34.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.005+0000 7f630dffb640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62fc06be30 0x7f62fc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.006+0000 7f630dffb640 1 -- 172.21.15.77:0/1055059564 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f631408bbd0 con 0x7f6318143940 2024-04-25T01:22:34.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.006+0000 7f631cc58640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62fc06be30 0x7f62fc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.006+0000 7f631cc58640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62fc06be30 0x7f62fc06e2f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6308007fd0 tx=0x7f6308007900 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:34.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.010+0000 7f630dffb640 1 -- 172.21.15.77:0/1055059564 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f631405a170 con 0x7f6318143940 2024-04-25T01:22:35.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.184+0000 7f631f6bd640 1 -- 172.21.15.77:0/1055059564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f631815cde0 con 0x7f6318143940 2024-04-25T01:22:35.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.185+0000 7f630dffb640 1 -- 172.21.15.77:0/1055059564 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18922 (secure 0 0 0) 0x7f6314059a70 con 0x7f6318143940 2024-04-25T01:22:35.167 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:35.167 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:26.287128+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:35.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 -- 172.21.15.77:0/1055059564 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62fc06be30 msgr2=0x7f62fc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62fc06be30 0x7f62fc06e2f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6308007fd0 tx=0x7f6308007900 comp rx=0 tx=0).stop 2024-04-25T01:22:35.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 -- 172.21.15.77:0/1055059564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6318143940 msgr2=0x7f6318156e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6318143940 0x7f6318156e90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6314007ef0 tx=0x7f631400e4b0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 -- 172.21.15.77:0/1055059564 shutdown_connections 2024-04-25T01:22:35.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f62fc06be30 0x7f62fc06e2f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f631815f400 0x7f631806d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6318143d20 0x7f6318156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 --2- 172.21.15.77:0/1055059564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6318143940 0x7f6318156e90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 -- 172.21.15.77:0/1055059564 >> 172.21.15.77:0/1055059564 conn(0x7f63180b1820 msgr2=0x7f63180b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:35.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 -- 172.21.15.77:0/1055059564 shutdown_connections 2024-04-25T01:22:35.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.187+0000 7f631f6bd640 1 -- 172.21.15.77:0/1055059564 wait complete. 2024-04-25T01:22:35.180 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:35.180 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:35.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.298+0000 7f99bd19e640 1 Processor -- start 2024-04-25T01:22:35.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.298+0000 7f99bd19e640 1 -- start start 2024-04-25T01:22:35.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99bd19e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99b815e0e0 0x7f99b80d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99bd19e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f99b80d49b0 0x7f99b80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99bd19e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99b815e4c0 con 0x7f99b80cf980 2024-04-25T01:22:35.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99bd19e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f99b80cf4f0 con 0x7f99b815e0e0 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99bd19e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f99b80cebc0 con 0x7f99b80d49b0 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99b7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99b815e0e0 0x7f99b80d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99b7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99b815e0e0 0x7f99b80d0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49278/0 (socket says 172.21.15.77:49278) 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99b7fff640 1 -- 172.21.15.77:0/88565406 learned_addr learned my addr 172.21.15.77:0/88565406 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99b77fe640 1 --2- 172.21.15.77:0/88565406 >> v2:172.21.15.121:3300/0 conn(0x7f99b80d49b0 0x7f99b80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99b7fff640 1 -- 172.21.15.77:0/88565406 >> v2:172.21.15.121:3300/0 conn(0x7f99b80d49b0 msgr2=0x7f99b80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99b7fff640 1 --2- 172.21.15.77:0/88565406 >> v2:172.21.15.121:3300/0 conn(0x7f99b80d49b0 0x7f99b80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99b7fff640 1 -- 172.21.15.77:0/88565406 >> v1:172.21.15.121:6789/0 conn(0x7f99b80cf980 legacy=0x7f99b80b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99b7fff640 1 -- 172.21.15.77:0/88565406 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b80ceea0 con 0x7f99b815e0e0 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.299+0000 7f99b77fe640 1 --2- 172.21.15.77:0/88565406 >> v2:172.21.15.121:3300/0 conn(0x7f99b80d49b0 0x7f99b80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.300+0000 7f99b7fff640 1 --2- 172.21.15.77:0/88565406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b815e0e0 0x7f99b80d0d10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f99b808e190 tx=0x7f99a802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c0b2eac743b6e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.300+0000 7f99b6ffd640 1 -- 172.21.15.77:0/88565406 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8037070 con 0x7f99b815e0e0 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.300+0000 7f99b6ffd640 1 -- 172.21.15.77:0/88565406 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99a8032440 con 0x7f99b815e0e0 2024-04-25T01:22:35.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.300+0000 7f99b6ffd640 1 -- 172.21.15.77:0/88565406 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8032810 con 0x7f99b815e0e0 2024-04-25T01:22:35.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.301+0000 7f99bd19e640 1 -- 172.21.15.77:0/88565406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b815e0e0 msgr2=0x7f99b80d0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.301+0000 7f99bd19e640 1 --2- 172.21.15.77:0/88565406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b815e0e0 0x7f99b80d0d10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f99b808e190 tx=0x7f99a802a230 comp rx=0 tx=0).stop 2024-04-25T01:22:35.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.301+0000 7f99bd19e640 1 -- 172.21.15.77:0/88565406 shutdown_connections 2024-04-25T01:22:35.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.301+0000 7f99bd19e640 1 --2- 172.21.15.77:0/88565406 >> v2:172.21.15.121:3300/0 conn(0x7f99b80d49b0 0x7f99b80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.301+0000 7f99bd19e640 1 --2- 172.21.15.77:0/88565406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b815e0e0 0x7f99b80d0d10 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.301+0000 7f99bd19e640 1 -- 172.21.15.77:0/88565406 >> 172.21.15.77:0/88565406 conn(0x7f99b80d3cd0 msgr2=0x7f99b80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:35.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.301+0000 7f99bd19e640 1 -- 172.21.15.77:0/88565406 shutdown_connections 2024-04-25T01:22:35.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.301+0000 7f99bd19e640 1 -- 172.21.15.77:0/88565406 wait complete. 2024-04-25T01:22:35.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.302+0000 7f99bd19e640 1 Processor -- start 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.302+0000 7f99bd19e640 1 -- start start 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bd19e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80cf980 0x7f99b80b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bd19e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99b80d49b0 0x7f99b80b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bd19e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b815e0e0 0x7f99b80c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bd19e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f99b80ce960 con 0x7f99b815e0e0 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bd19e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f99b80ce7e0 con 0x7f99b80cf980 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bd19e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f99b80d1250 con 0x7f99b80d49b0 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bc99d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b815e0e0 0x7f99b80c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bc99d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b815e0e0 0x7f99b80c9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49292/0 (socket says 172.21.15.77:49292) 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bc99d640 1 -- 172.21.15.77:0/1245348762 learned_addr learned my addr 172.21.15.77:0/1245348762 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99b7fff640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80cf980 0x7f99b80b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99b77fe640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99b80d49b0 0x7f99b80b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bc99d640 1 -- 172.21.15.77:0/1245348762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99b80d49b0 msgr2=0x7f99b80b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bc99d640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99b80d49b0 0x7f99b80b31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bc99d640 1 -- 172.21.15.77:0/1245348762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80cf980 msgr2=0x7f99b80b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bc99d640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80cf980 0x7f99b80b4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bc99d640 1 -- 172.21.15.77:0/1245348762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b80b5340 con 0x7f99b815e0e0 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.303+0000 7f99bc99d640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b815e0e0 0x7f99b80c9f70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f99b000c780 tx=0x7f99b000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.304+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1245348762 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b0004b60 con 0x7f99b815e0e0 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.304+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1245348762 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b0004d00 con 0x7f99b815e0e0 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.304+0000 7f99bd19e640 1 -- 172.21.15.77:0/1245348762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b80b5620 con 0x7f99b815e0e0 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.304+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1245348762 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b000aa90 con 0x7f99b815e0e0 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.304+0000 7f99bd19e640 1 -- 172.21.15.77:0/1245348762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b80b5be0 con 0x7f99b815e0e0 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.305+0000 7f99bd19e640 1 -- 172.21.15.77:0/1245348762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b8075350 con 0x7f99b815e0e0 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.305+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1245348762 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f99b000ac30 con 0x7f99b815e0e0 2024-04-25T01:22:35.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.305+0000 7f99b4ff9640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f99a406be30 0x7f99a406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.306+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1245348762 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f99b008b570 con 0x7f99b815e0e0 2024-04-25T01:22:35.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.306+0000 7f99b7fff640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f99a406be30 0x7f99a406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.306+0000 7f99b7fff640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f99a406be30 0x7f99a406e2f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f99a8034000 tx=0x7f99a802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:35.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.310+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1245348762 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f99b0020070 con 0x7f99b815e0e0 2024-04-25T01:22:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.482+0000 7f99bd19e640 1 -- 172.21.15.77:0/1245348762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f99b8074b00 con 0x7f99b815e0e0 2024-04-25T01:22:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.483+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1245348762 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f99b0059b10 con 0x7f99b815e0e0 2024-04-25T01:22:35.464 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:35.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.484+0000 7f99bd19e640 1 -- 172.21.15.77:0/1245348762 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f99a406be30 msgr2=0x7f99a406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.484+0000 7f99bd19e640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f99a406be30 0x7f99a406e2f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f99a8034000 tx=0x7f99a802e000 comp rx=0 tx=0).stop 2024-04-25T01:22:35.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.484+0000 7f99bd19e640 1 -- 172.21.15.77:0/1245348762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b815e0e0 msgr2=0x7f99b80c9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.484+0000 7f99bd19e640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b815e0e0 0x7f99b80c9f70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f99b000c780 tx=0x7f99b000cc40 comp rx=0 tx=0).stop 2024-04-25T01:22:35.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.485+0000 7f99bd19e640 1 -- 172.21.15.77:0/1245348762 shutdown_connections 2024-04-25T01:22:35.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.485+0000 7f99bd19e640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f99a406be30 0x7f99a406e2f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.485+0000 7f99bd19e640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b815e0e0 0x7f99b80c9f70 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.485+0000 7f99bd19e640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99b80d49b0 0x7f99b80b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.485+0000 7f99bd19e640 1 --2- 172.21.15.77:0/1245348762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80cf980 0x7f99b80b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.485+0000 7f99bd19e640 1 -- 172.21.15.77:0/1245348762 >> 172.21.15.77:0/1245348762 conn(0x7f99b80d3cd0 msgr2=0x7f99b815ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:35.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.485+0000 7f99bd19e640 1 -- 172.21.15.77:0/1245348762 shutdown_connections 2024-04-25T01:22:35.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.485+0000 7f99bd19e640 1 -- 172.21.15.77:0/1245348762 wait complete. 2024-04-25T01:22:35.477 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:35.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.595+0000 7ffa2e3c4640 1 Processor -- start 2024-04-25T01:22:35.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.595+0000 7ffa2e3c4640 1 -- start start 2024-04-25T01:22:35.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa2e3c4640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ffa280b42c0 0x7ffa2806cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa2e3c4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa280d0a90 con 0x7ffa28161df0 2024-04-25T01:22:35.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa2e3c4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa280d0d70 con 0x7ffa280b2920 2024-04-25T01:22:35.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa2e3c4640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ffa28162d00 con 0x7ffa280b42c0 2024-04-25T01:22:35.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa27fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ffa28161df0 0x7ffa281621d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57506/0 (socket says 172.21.15.77:57506) 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa27fff640 1 -- 172.21.15.77:0/3923640908 learned_addr learned my addr 172.21.15.77:0/3923640908 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339153447 0 0) 0x7ffa280d0a90 con 0x7ffa28161df0 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa10003520 con 0x7ffa28161df0 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa2c961640 1 --2- 172.21.15.77:0/3923640908 >> v2:172.21.15.121:3300/0 conn(0x7ffa280b42c0 0x7ffa2806cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1618388524 0 0) 0x7ffa10003520 con 0x7ffa28161df0 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 >> v2:172.21.15.121:3300/0 conn(0x7ffa280b42c0 msgr2=0x7ffa2806cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 --2- 172.21.15.77:0/3923640908 >> v2:172.21.15.121:3300/0 conn(0x7ffa280b42c0 0x7ffa2806cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 >> v1:172.21.15.121:6789/0 conn(0x7ffa280b2920 legacy=0x7ffa2806c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa280d1050 con 0x7ffa28161df0 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7ffa1c001670 con 0x7ffa28161df0 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 >> v1:172.21.15.77:6789/0 conn(0x7ffa28161df0 legacy=0x7ffa281621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 --2- 172.21.15.77:0/3923640908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa100039b0 0x7ffa10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 --2- 172.21.15.77:0/3923640908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa10004800 0x7ffa10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 --2- 172.21.15.77:0/3923640908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa100053c0 0x7ffa1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa28162d00 con 0x7ffa100053c0 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1000c250 con 0x7ffa100039b0 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.596+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffa1000c450 con 0x7ffa10004800 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.597+0000 7ffa2c961640 1 --2- 172.21.15.77:0/3923640908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa100053c0 0x7ffa1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.597+0000 7ffa27fff640 1 --2- 172.21.15.77:0/3923640908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa100039b0 0x7ffa10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.597+0000 7ffa277fe640 1 --2- 172.21.15.77:0/3923640908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa10004800 0x7ffa10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.597+0000 7ffa2c961640 1 -- 172.21.15.77:0/3923640908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa10004800 msgr2=0x7ffa10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.597+0000 7ffa2c961640 1 --2- 172.21.15.77:0/3923640908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa10004800 0x7ffa10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.597+0000 7ffa2c961640 1 -- 172.21.15.77:0/3923640908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa100039b0 msgr2=0x7ffa10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.597+0000 7ffa2c961640 1 --2- 172.21.15.77:0/3923640908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa100039b0 0x7ffa10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.597+0000 7ffa2c961640 1 -- 172.21.15.77:0/3923640908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa10003520 con 0x7ffa100053c0 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.597+0000 7ffa2c961640 1 -- 172.21.15.77:0/3923640908 reap_dead start 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.597+0000 7ffa2c961640 1 --2- 172.21.15.77:0/3923640908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa100053c0 0x7ffa1000b9c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffa20007360 tx=0x7ffa10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10c954c7e0c0000 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:35.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.598+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20009be0 con 0x7ffa100053c0 2024-04-25T01:22:35.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.598+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa20003bd0 con 0x7ffa100053c0 2024-04-25T01:22:35.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.598+0000 7ffa26ffd640 1 -- 172.21.15.77:0/3923640908 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20004020 con 0x7ffa100053c0 2024-04-25T01:22:35.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.598+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/3923640908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa100053c0 msgr2=0x7ffa1000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.598+0000 7ffa2e3c4640 1 --2- 172.21.15.77:0/3923640908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa100053c0 0x7ffa1000b9c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffa20007360 tx=0x7ffa10005830 comp rx=0 tx=0).stop 2024-04-25T01:22:35.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.598+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/3923640908 shutdown_connections 2024-04-25T01:22:35.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.598+0000 7ffa2e3c4640 1 --2- 172.21.15.77:0/3923640908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa100053c0 0x7ffa1000b9c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.598+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/3923640908 >> 172.21.15.77:0/3923640908 conn(0x7ffa280b37a0 msgr2=0x7ffa280b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:35.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.598+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/3923640908 shutdown_connections 2024-04-25T01:22:35.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.598+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/3923640908 wait complete. 2024-04-25T01:22:35.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.599+0000 7ffa2e3c4640 1 Processor -- start 2024-04-25T01:22:35.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa2e3c4640 1 -- start start 2024-04-25T01:22:35.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa2e3c4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa100053c0 0x7ffa280b0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa2e3c4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280b6420 0x7ffa280b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa2e3c4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa2814efb0 0x7ffa280c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa2e3c4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa280aec00 con 0x7ffa280b6420 2024-04-25T01:22:35.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa2e3c4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffa280aea80 con 0x7ffa100053c0 2024-04-25T01:22:35.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa2e3c4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffa281621d0 con 0x7ffa2814efb0 2024-04-25T01:22:35.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa277fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280b6420 0x7ffa280b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa277fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280b6420 0x7ffa280b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49310/0 (socket says 172.21.15.77:49310) 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa277fe640 1 -- 172.21.15.77:0/810175938 learned_addr learned my addr 172.21.15.77:0/810175938 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa27fff640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa100053c0 0x7ffa280b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.600+0000 7ffa2c961640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa2814efb0 0x7ffa280c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.601+0000 7ffa277fe640 1 -- 172.21.15.77:0/810175938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa2814efb0 msgr2=0x7ffa280c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.601+0000 7ffa277fe640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa2814efb0 0x7ffa280c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.601+0000 7ffa277fe640 1 -- 172.21.15.77:0/810175938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa100053c0 msgr2=0x7ffa280b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.601+0000 7ffa277fe640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa100053c0 0x7ffa280b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.601+0000 7ffa277fe640 1 -- 172.21.15.77:0/810175938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa280b0810 con 0x7ffa280b6420 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.601+0000 7ffa277fe640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280b6420 0x7ffa280b18b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800f570 tx=0x7ffa1800fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.601+0000 7ffa24ff9640 1 -- 172.21.15.77:0/810175938 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa180052e0 con 0x7ffa280b6420 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.601+0000 7ffa24ff9640 1 -- 172.21.15.77:0/810175938 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa18018070 con 0x7ffa280b6420 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.601+0000 7ffa24ff9640 1 -- 172.21.15.77:0/810175938 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18003830 con 0x7ffa280b6420 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.601+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/810175938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa280b0af0 con 0x7ffa280b6420 2024-04-25T01:22:35.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.601+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/810175938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa280b10b0 con 0x7ffa280b6420 2024-04-25T01:22:35.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.602+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/810175938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa28068180 con 0x7ffa280b6420 2024-04-25T01:22:35.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.602+0000 7ffa24ff9640 1 -- 172.21.15.77:0/810175938 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ffa180039d0 con 0x7ffa280b6420 2024-04-25T01:22:35.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.603+0000 7ffa24ff9640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa1406be30 0x7ffa1406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.603+0000 7ffa27fff640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa1406be30 0x7ffa1406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.603+0000 7ffa24ff9640 1 -- 172.21.15.77:0/810175938 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7ffa1808aee0 con 0x7ffa280b6420 2024-04-25T01:22:35.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.603+0000 7ffa27fff640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa1406be30 0x7ffa1406e2f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c001eb0 tx=0x7ffa1c000fa0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:35.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.607+0000 7ffa24ff9640 1 -- 172.21.15.77:0/810175938 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffa18058e30 con 0x7ffa280b6420 2024-04-25T01:22:35.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.786+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/810175938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffa280d2ac0 con 0x7ffa280b6420 2024-04-25T01:22:35.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.786+0000 7ffa24ff9640 1 -- 172.21.15.77:0/810175938 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7ffa1805d130 con 0x7ffa280b6420 2024-04-25T01:22:35.768 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:35.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/810175938 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa1406be30 msgr2=0x7ffa1406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa1406be30 0x7ffa1406e2f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c001eb0 tx=0x7ffa1c000fa0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/810175938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280b6420 msgr2=0x7ffa280b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280b6420 0x7ffa280b18b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800f570 tx=0x7ffa1800fa30 comp rx=0 tx=0).stop 2024-04-25T01:22:35.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/810175938 shutdown_connections 2024-04-25T01:22:35.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa1406be30 0x7ffa1406e2f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280b6420 0x7ffa280b18b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa2814efb0 0x7ffa280c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 --2- 172.21.15.77:0/810175938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa100053c0 0x7ffa280b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/810175938 >> 172.21.15.77:0/810175938 conn(0x7ffa280b37a0 msgr2=0x7ffa28070370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:35.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/810175938 shutdown_connections 2024-04-25T01:22:35.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.788+0000 7ffa2e3c4640 1 -- 172.21.15.77:0/810175938 wait complete. 2024-04-25T01:22:35.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:35.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.899+0000 7f4b4bb90640 1 Processor -- start 2024-04-25T01:22:35.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.899+0000 7f4b4bb90640 1 -- start start 2024-04-25T01:22:35.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4bb90640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4b440b47d0 0x7f4b4415b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4bb90640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4b440b37c0 0x7f4b4415ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4bb90640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b4415bfa0 con 0x7f4b44158a70 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4bb90640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f4b4415c210 con 0x7f4b440b47d0 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4bb90640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4b440b2ce0 con 0x7f4b440b37c0 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4992c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4b44158a70 0x7f4b440b40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57512/0 (socket says 172.21.15.77:57512) 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4992c640 1 -- 172.21.15.77:0/2874718494 learned_addr learned my addr 172.21.15.77:0/2874718494 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4912b640 1 --2- 172.21.15.77:0/2874718494 >> v2:172.21.15.77:3300/0 conn(0x7f4b440b47d0 0x7f4b4415b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4892a640 1 -- 172.21.15.77:0/2874718494 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2935091984 0 0) 0x7f4b4415bfa0 con 0x7f4b44158a70 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4892a640 1 -- 172.21.15.77:0/2874718494 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b30003520 con 0x7f4b44158a70 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4912b640 1 -- 172.21.15.77:0/2874718494 >> v2:172.21.15.121:3300/0 conn(0x7f4b440b37c0 msgr2=0x7f4b4415ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4912b640 1 --2- 172.21.15.77:0/2874718494 >> v2:172.21.15.121:3300/0 conn(0x7f4b440b37c0 0x7f4b4415ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4912b640 1 -- 172.21.15.77:0/2874718494 >> v1:172.21.15.77:6789/0 conn(0x7f4b44158a70 legacy=0x7f4b440b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.900+0000 7f4b4912b640 1 -- 172.21.15.77:0/2874718494 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b440b2fc0 con 0x7f4b440b47d0 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.901+0000 7f4b4912b640 1 --2- 172.21.15.77:0/2874718494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440b47d0 0x7f4b4415b520 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4b340097a0 tx=0x7f4b3402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f70543f71aaf68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.901+0000 7f4b4892a640 1 -- 172.21.15.77:0/2874718494 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34037070 con 0x7f4b440b47d0 2024-04-25T01:22:35.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.901+0000 7f4b4892a640 1 -- 172.21.15.77:0/2874718494 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b34032440 con 0x7f4b440b47d0 2024-04-25T01:22:35.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.901+0000 7f4b4892a640 1 -- 172.21.15.77:0/2874718494 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34032810 con 0x7f4b440b47d0 2024-04-25T01:22:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.902+0000 7f4b4bb90640 1 -- 172.21.15.77:0/2874718494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440b47d0 msgr2=0x7f4b4415b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.902+0000 7f4b4bb90640 1 --2- 172.21.15.77:0/2874718494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440b47d0 0x7f4b4415b520 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4b340097a0 tx=0x7f4b3402a1e0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.902+0000 7f4b4bb90640 1 -- 172.21.15.77:0/2874718494 shutdown_connections 2024-04-25T01:22:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.902+0000 7f4b4bb90640 1 --2- 172.21.15.77:0/2874718494 >> v2:172.21.15.121:3300/0 conn(0x7f4b440b37c0 0x7f4b4415ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.902+0000 7f4b4bb90640 1 --2- 172.21.15.77:0/2874718494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440b47d0 0x7f4b4415b520 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.902+0000 7f4b4bb90640 1 -- 172.21.15.77:0/2874718494 >> 172.21.15.77:0/2874718494 conn(0x7f4b4406b720 msgr2=0x7f4b440b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.902+0000 7f4b4bb90640 1 -- 172.21.15.77:0/2874718494 shutdown_connections 2024-04-25T01:22:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.902+0000 7f4b4bb90640 1 -- 172.21.15.77:0/2874718494 wait complete. 2024-04-25T01:22:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.903+0000 7f4b4bb90640 1 Processor -- start 2024-04-25T01:22:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.903+0000 7f4b4bb90640 1 -- start start 2024-04-25T01:22:35.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4bb90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440b37c0 0x7f4b4415f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4bb90640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b440b47d0 0x7f4b440ce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4bb90640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44158a70 0x7f4b4413f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4bb90640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4406c9f0 con 0x7f4b440b37c0 2024-04-25T01:22:35.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4bb90640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44158e50 con 0x7f4b44158a70 2024-04-25T01:22:35.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4bb90640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4b4406cb70 con 0x7f4b440b47d0 2024-04-25T01:22:35.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4992c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440b37c0 0x7f4b4415f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4992c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440b37c0 0x7f4b4415f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49328/0 (socket says 172.21.15.77:49328) 2024-04-25T01:22:35.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4992c640 1 -- 172.21.15.77:0/3317456063 learned_addr learned my addr 172.21.15.77:0/3317456063 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4a12d640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44158a70 0x7f4b4413f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4912b640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b440b47d0 0x7f4b440ce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4992c640 1 -- 172.21.15.77:0/3317456063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b440b47d0 msgr2=0x7f4b440ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4992c640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b440b47d0 0x7f4b440ce6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4992c640 1 -- 172.21.15.77:0/3317456063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44158a70 msgr2=0x7f4b4413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4992c640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44158a70 0x7f4b4413f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4992c640 1 -- 172.21.15.77:0/3317456063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b440cd290 con 0x7f4b440b37c0 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4a12d640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44158a70 0x7f4b4413f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4912b640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b440b47d0 0x7f4b440ce6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4992c640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440b37c0 0x7f4b4415f5e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4b4000caa0 tx=0x7f4b4000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b3a7fc640 1 -- 172.21.15.77:0/3317456063 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b40017070 con 0x7f4b440b37c0 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b3a7fc640 1 -- 172.21.15.77:0/3317456063 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b4000ac00 con 0x7f4b440b37c0 2024-04-25T01:22:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4bb90640 1 -- 172.21.15.77:0/3317456063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b440cd570 con 0x7f4b440b37c0 2024-04-25T01:22:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b3a7fc640 1 -- 172.21.15.77:0/3317456063 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b40012400 con 0x7f4b440b37c0 2024-04-25T01:22:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.904+0000 7f4b4bb90640 1 -- 172.21.15.77:0/3317456063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b440cdb30 con 0x7f4b440b37c0 2024-04-25T01:22:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.905+0000 7f4b4bb90640 1 -- 172.21.15.77:0/3317456063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b44068fb0 con 0x7f4b440b37c0 2024-04-25T01:22:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.906+0000 7f4b3a7fc640 1 -- 172.21.15.77:0/3317456063 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4b4001a070 con 0x7f4b440b37c0 2024-04-25T01:22:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.906+0000 7f4b3a7fc640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b2806be60 0x7f4b2806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.906+0000 7f4b3a7fc640 1 -- 172.21.15.77:0/3317456063 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f4b4008b000 con 0x7f4b440b37c0 2024-04-25T01:22:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.906+0000 7f4b4912b640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b2806be60 0x7f4b2806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.907+0000 7f4b4912b640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b2806be60 0x7f4b2806e320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4b34034000 tx=0x7f4b3402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:35.910+0000 7f4b3a7fc640 1 -- 172.21.15.77:0/3317456063 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4b400595a0 con 0x7f4b440b37c0 2024-04-25T01:22:36.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.083+0000 7f4b4bb90640 1 -- 172.21.15.77:0/3317456063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4b440b3c70 con 0x7f4b440b37c0 2024-04-25T01:22:36.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.083+0000 7f4b3a7fc640 1 -- 172.21.15.77:0/3317456063 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f4b40058ea0 con 0x7f4b440b37c0 2024-04-25T01:22:36.065 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:36.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 -- 172.21.15.77:0/3317456063 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b2806be60 msgr2=0x7f4b2806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b2806be60 0x7f4b2806e320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4b34034000 tx=0x7f4b3402e000 comp rx=0 tx=0).stop 2024-04-25T01:22:36.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 -- 172.21.15.77:0/3317456063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440b37c0 msgr2=0x7f4b4415f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440b37c0 0x7f4b4415f5e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4b4000caa0 tx=0x7f4b4000cf60 comp rx=0 tx=0).stop 2024-04-25T01:22:36.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 -- 172.21.15.77:0/3317456063 shutdown_connections 2024-04-25T01:22:36.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4b2806be60 0x7f4b2806e320 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44158a70 0x7f4b4413f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b440b47d0 0x7f4b440ce6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 --2- 172.21.15.77:0/3317456063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440b37c0 0x7f4b4415f5e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 -- 172.21.15.77:0/3317456063 >> 172.21.15.77:0/3317456063 conn(0x7f4b4406b720 msgr2=0x7f4b44071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:36.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 -- 172.21.15.77:0/3317456063 shutdown_connections 2024-04-25T01:22:36.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.085+0000 7f4b4bb90640 1 -- 172.21.15.77:0/3317456063 wait complete. 2024-04-25T01:22:36.077 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:36.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.196+0000 7f1cc8763640 1 Processor -- start 2024-04-25T01:22:36.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.196+0000 7f1cc8763640 1 -- start start 2024-04-25T01:22:36.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc8763640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1cc00722e0 0x7f1cc006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc8763640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc015fc40 0x7f1cc01600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc8763640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1cc00cedc0 0x7f1cc00cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc8763640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f1cc0160aa0 con 0x7f1cc00722e0 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc8763640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1cc006da40 con 0x7f1cc00cedc0 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc8763640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1cc006cec0 con 0x7f1cc015fc40 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc64ff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1cc00722e0 0x7f1cc006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc64ff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1cc00722e0 0x7f1cc006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49332/0 (socket says 172.21.15.77:49332) 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc64ff640 1 -- 172.21.15.77:0/794931210 learned_addr learned my addr 172.21.15.77:0/794931210 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc5cfe640 1 --2- 172.21.15.77:0/794931210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc015fc40 0x7f1cc01600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc64ff640 1 -- 172.21.15.77:0/794931210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc015fc40 msgr2=0x7f1cc01600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc64ff640 1 --2- 172.21.15.77:0/794931210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc015fc40 0x7f1cc01600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc64ff640 1 -- 172.21.15.77:0/794931210 >> v2:172.21.15.121:3300/0 conn(0x7f1cc00cedc0 msgr2=0x7f1cc00cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc6d00640 1 --2- 172.21.15.77:0/794931210 >> v2:172.21.15.121:3300/0 conn(0x7f1cc00cedc0 0x7f1cc00cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc64ff640 1 --2- 172.21.15.77:0/794931210 >> v2:172.21.15.121:3300/0 conn(0x7f1cc00cedc0 0x7f1cc00cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc64ff640 1 -- 172.21.15.77:0/794931210 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc00cf790 con 0x7f1cc00722e0 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc5cfe640 1 --2- 172.21.15.77:0/794931210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc015fc40 0x7f1cc01600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.197+0000 7f1cc6d00640 1 --2- 172.21.15.77:0/794931210 >> v2:172.21.15.121:3300/0 conn(0x7f1cc00cedc0 0x7f1cc00cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.198+0000 7f1cc64ff640 1 --2- 172.21.15.77:0/794931210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc00722e0 0x7f1cc006ded0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00af5e0 tx=0x7f1cbc02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c71e5a0e2aef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.198+0000 7f1cc54fd640 1 -- 172.21.15.77:0/794931210 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc037070 con 0x7f1cc00722e0 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.198+0000 7f1cc54fd640 1 -- 172.21.15.77:0/794931210 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cbc032440 con 0x7f1cc00722e0 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.198+0000 7f1cc54fd640 1 -- 172.21.15.77:0/794931210 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc032810 con 0x7f1cc00722e0 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.199+0000 7f1cc8763640 1 -- 172.21.15.77:0/794931210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc00722e0 msgr2=0x7f1cc006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.199+0000 7f1cc8763640 1 --2- 172.21.15.77:0/794931210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc00722e0 0x7f1cc006ded0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00af5e0 tx=0x7f1cbc02a140 comp rx=0 tx=0).stop 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.199+0000 7f1cc8763640 1 -- 172.21.15.77:0/794931210 shutdown_connections 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.199+0000 7f1cc8763640 1 --2- 172.21.15.77:0/794931210 >> v2:172.21.15.121:3300/0 conn(0x7f1cc00cedc0 0x7f1cc00cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.199+0000 7f1cc8763640 1 --2- 172.21.15.77:0/794931210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc015fc40 0x7f1cc01600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.199+0000 7f1cc8763640 1 --2- 172.21.15.77:0/794931210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc00722e0 0x7f1cc006ded0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.199+0000 7f1cc8763640 1 -- 172.21.15.77:0/794931210 >> 172.21.15.77:0/794931210 conn(0x7f1cc006b720 msgr2=0x7f1cc00b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:36.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.199+0000 7f1cc8763640 1 -- 172.21.15.77:0/794931210 shutdown_connections 2024-04-25T01:22:36.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.199+0000 7f1cc8763640 1 -- 172.21.15.77:0/794931210 wait complete. 2024-04-25T01:22:36.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.200+0000 7f1cc8763640 1 Processor -- start 2024-04-25T01:22:36.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.200+0000 7f1cc8763640 1 -- start start 2024-04-25T01:22:36.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc8763640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc00722e0 0x7f1cc0157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc8763640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc00cedc0 0x7f1cc0155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc8763640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc015fc40 0x7f1cc0157c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc8763640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc006d3f0 con 0x7f1cc015fc40 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc8763640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc006e410 con 0x7f1cc00722e0 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc8763640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1cc006d6d0 con 0x7f1cc00cedc0 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc6d00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc015fc40 0x7f1cc0157c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc6d00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc015fc40 0x7f1cc0157c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49346/0 (socket says 172.21.15.77:49346) 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc6d00640 1 -- 172.21.15.77:0/498363165 learned_addr learned my addr 172.21.15.77:0/498363165 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc5cfe640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc00cedc0 0x7f1cc0155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc64ff640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc00722e0 0x7f1cc0157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc6d00640 1 -- 172.21.15.77:0/498363165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc00cedc0 msgr2=0x7f1cc0155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc6d00640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc00cedc0 0x7f1cc0155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc6d00640 1 -- 172.21.15.77:0/498363165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc00722e0 msgr2=0x7f1cc0157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc6d00640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc00722e0 0x7f1cc0157460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc6d00640 1 -- 172.21.15.77:0/498363165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc01598e0 con 0x7f1cc015fc40 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc6d00640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc015fc40 0x7f1cc0157c30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800cef0 tx=0x7f1cb800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cb6ffd640 1 -- 172.21.15.77:0/498363165 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb801a070 con 0x7f1cc015fc40 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cb6ffd640 1 -- 172.21.15.77:0/498363165 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cb800d840 con 0x7f1cc015fc40 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc8763640 1 -- 172.21.15.77:0/498363165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc0159bc0 con 0x7f1cc015fc40 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cb6ffd640 1 -- 172.21.15.77:0/498363165 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb800dc30 con 0x7f1cc015fc40 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.201+0000 7f1cc8763640 1 -- 172.21.15.77:0/498363165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc01589b0 con 0x7f1cc015fc40 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.202+0000 7f1cc8763640 1 -- 172.21.15.77:0/498363165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc0068fb0 con 0x7f1cc015fc40 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.203+0000 7f1cb6ffd640 1 -- 172.21.15.77:0/498363165 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1cb801d070 con 0x7f1cc015fc40 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.203+0000 7f1cb6ffd640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ca406be30 0x7f1ca406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.203+0000 7f1cb6ffd640 1 -- 172.21.15.77:0/498363165 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f1cb808ba70 con 0x7f1cc015fc40 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.203+0000 7f1cc64ff640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ca406be30 0x7f1ca406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.204+0000 7f1cc64ff640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ca406be30 0x7f1ca406e2f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc034000 tx=0x7f1cbc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:36.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.208+0000 7f1cb6ffd640 1 -- 172.21.15.77:0/498363165 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1cb805a0a0 con 0x7f1cc015fc40 2024-04-25T01:22:36.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.380+0000 7f1cc8763640 1 -- 172.21.15.77:0/498363165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1cc0072c90 con 0x7f1cc015fc40 2024-04-25T01:22:36.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.380+0000 7f1cb6ffd640 1 -- 172.21.15.77:0/498363165 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f1cb80599a0 con 0x7f1cc015fc40 2024-04-25T01:22:36.362 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:36.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.382+0000 7f1cc8763640 1 -- 172.21.15.77:0/498363165 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ca406be30 msgr2=0x7f1ca406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.382+0000 7f1cc8763640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ca406be30 0x7f1ca406e2f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc034000 tx=0x7f1cbc02e000 comp rx=0 tx=0).stop 2024-04-25T01:22:36.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.382+0000 7f1cc8763640 1 -- 172.21.15.77:0/498363165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc015fc40 msgr2=0x7f1cc0157c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.382+0000 7f1cc8763640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc015fc40 0x7f1cc0157c30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800cef0 tx=0x7f1cb800a710 comp rx=0 tx=0).stop 2024-04-25T01:22:36.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.382+0000 7f1cc8763640 1 -- 172.21.15.77:0/498363165 shutdown_connections 2024-04-25T01:22:36.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.382+0000 7f1cc8763640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ca406be30 0x7f1ca406e2f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.382+0000 7f1cc8763640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc015fc40 0x7f1cc0157c30 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.382+0000 7f1cc8763640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc00cedc0 0x7f1cc0155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.382+0000 7f1cc8763640 1 --2- 172.21.15.77:0/498363165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc00722e0 0x7f1cc0157460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.382+0000 7f1cc8763640 1 -- 172.21.15.77:0/498363165 >> 172.21.15.77:0/498363165 conn(0x7f1cc006b720 msgr2=0x7f1cc015e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:36.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.383+0000 7f1cc8763640 1 -- 172.21.15.77:0/498363165 shutdown_connections 2024-04-25T01:22:36.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.383+0000 7f1cc8763640 1 -- 172.21.15.77:0/498363165 wait complete. 2024-04-25T01:22:36.375 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:22:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.491+0000 7fb044c15640 1 Processor -- start 2024-04-25T01:22:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.491+0000 7fb044c15640 1 -- start start 2024-04-25T01:22:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.492+0000 7fb044c15640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb040143d20 0x7fb04015ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.492+0000 7fb044c15640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb040070e60 con 0x7fb040143940 2024-04-25T01:22:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.492+0000 7fb044c15640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb040071140 con 0x7fb04006ed60 2024-04-25T01:22:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.492+0000 7fb044c15640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fb040071420 con 0x7fb040143d20 2024-04-25T01:22:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.492+0000 7fb037fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb040143d20 0x7fb04015ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.492+0000 7fb03e59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb040143940 0x7fb04006e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57518/0 (socket says 172.21.15.77:57518) 2024-04-25T01:22:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.492+0000 7fb03e59c640 1 -- 172.21.15.77:0/3690134137 learned_addr learned my addr 172.21.15.77:0/3690134137 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.492+0000 7fb037fff640 1 -- 172.21.15.77:0/3690134137 >> v1:172.21.15.121:6789/0 conn(0x7fb04006ed60 legacy=0x7fb04015f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.492+0000 7fb037fff640 1 -- 172.21.15.77:0/3690134137 >> v1:172.21.15.77:6789/0 conn(0x7fb040143940 legacy=0x7fb04006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.492+0000 7fb037fff640 1 -- 172.21.15.77:0/3690134137 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb040071700 con 0x7fb040143d20 2024-04-25T01:22:36.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.493+0000 7fb037fff640 1 --2- 172.21.15.77:0/3690134137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb040143d20 0x7fb04015ecf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb0280097d0 tx=0x7fb02802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fcadf6a00f97d25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:36.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.493+0000 7fb03dd9b640 1 -- 172.21.15.77:0/3690134137 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb028037070 con 0x7fb040143d20 2024-04-25T01:22:36.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.493+0000 7fb03dd9b640 1 -- 172.21.15.77:0/3690134137 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb028032440 con 0x7fb040143d20 2024-04-25T01:22:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.493+0000 7fb03dd9b640 1 -- 172.21.15.77:0/3690134137 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb028032810 con 0x7fb040143d20 2024-04-25T01:22:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.493+0000 7fb044c15640 1 -- 172.21.15.77:0/3690134137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb040143d20 msgr2=0x7fb04015ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.493+0000 7fb044c15640 1 --2- 172.21.15.77:0/3690134137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb040143d20 0x7fb04015ecf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb0280097d0 tx=0x7fb02802a210 comp rx=0 tx=0).stop 2024-04-25T01:22:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.493+0000 7fb044c15640 1 -- 172.21.15.77:0/3690134137 shutdown_connections 2024-04-25T01:22:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.493+0000 7fb044c15640 1 --2- 172.21.15.77:0/3690134137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb040143d20 0x7fb04015ecf0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.493+0000 7fb044c15640 1 -- 172.21.15.77:0/3690134137 >> 172.21.15.77:0/3690134137 conn(0x7fb0400b1820 msgr2=0x7fb0400b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.494+0000 7fb044c15640 1 -- 172.21.15.77:0/3690134137 shutdown_connections 2024-04-25T01:22:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.494+0000 7fb044c15640 1 -- 172.21.15.77:0/3690134137 wait complete. 2024-04-25T01:22:36.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.495+0000 7fb044c15640 1 Processor -- start 2024-04-25T01:22:36.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.495+0000 7fb044c15640 1 -- start start 2024-04-25T01:22:36.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb044c15640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb04006ed60 0x7fb04015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb044c15640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb040143940 0x7fb04015b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb044c15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb040143d20 0x7fb04015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb044c15640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb040165d30 con 0x7fb040143d20 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb044c15640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb040072e80 con 0x7fb040143940 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb044c15640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb040165eb0 con 0x7fb04006ed60 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb03ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb040143d20 0x7fb04015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb03ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb040143d20 0x7fb04015b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49366/0 (socket says 172.21.15.77:49366) 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb03ed9d640 1 -- 172.21.15.77:0/995384487 learned_addr learned my addr 172.21.15.77:0/995384487 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb03ed9d640 1 -- 172.21.15.77:0/995384487 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb04006ed60 msgr2=0x7fb04015dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb037fff640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb040143940 0x7fb04015b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb03e59c640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb04006ed60 0x7fb04015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb03ed9d640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb04006ed60 0x7fb04015dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb03ed9d640 1 -- 172.21.15.77:0/995384487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb040143940 msgr2=0x7fb04015b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb03ed9d640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb040143940 0x7fb04015b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb03ed9d640 1 -- 172.21.15.77:0/995384487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04013f7f0 con 0x7fb040143d20 2024-04-25T01:22:36.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb03ed9d640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb040143d20 0x7fb04015b8f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb03800c780 tx=0x7fb03800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:36.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb036ffd640 1 -- 172.21.15.77:0/995384487 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038017070 con 0x7fb040143d20 2024-04-25T01:22:36.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb036ffd640 1 -- 172.21.15.77:0/995384487 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb038003da0 con 0x7fb040143d20 2024-04-25T01:22:36.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb044c15640 1 -- 172.21.15.77:0/995384487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04013fad0 con 0x7fb040143d20 2024-04-25T01:22:36.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb036ffd640 1 -- 172.21.15.77:0/995384487 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0380052b0 con 0x7fb040143d20 2024-04-25T01:22:36.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.496+0000 7fb044c15640 1 -- 172.21.15.77:0/995384487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0401370d0 con 0x7fb040143d20 2024-04-25T01:22:36.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.497+0000 7fb044c15640 1 -- 172.21.15.77:0/995384487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb04013fec0 con 0x7fb040143d20 2024-04-25T01:22:36.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.498+0000 7fb036ffd640 1 -- 172.21.15.77:0/995384487 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb03801a070 con 0x7fb040143d20 2024-04-25T01:22:36.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.498+0000 7fb036ffd640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb02006be30 0x7fb02006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.498+0000 7fb036ffd640 1 -- 172.21.15.77:0/995384487 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fb03808a8a0 con 0x7fb040143d20 2024-04-25T01:22:36.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.498+0000 7fb03e59c640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb02006be30 0x7fb02006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.499+0000 7fb03e59c640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb02006be30 0x7fb02006e2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb030000f60 tx=0x7fb030009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:36.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.502+0000 7fb036ffd640 1 -- 172.21.15.77:0/995384487 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb038058e40 con 0x7fb040143d20 2024-04-25T01:22:36.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.689+0000 7fb044c15640 1 -- 172.21.15.77:0/995384487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb040068fb0 con 0x7fb040143d20 2024-04-25T01:22:36.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.690+0000 7fb036ffd640 1 -- 172.21.15.77:0/995384487 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7fb038058740 con 0x7fb040143d20 2024-04-25T01:22:36.671 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-04-25T01:22:36.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 -- 172.21.15.77:0/995384487 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb02006be30 msgr2=0x7fb02006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb02006be30 0x7fb02006e2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb030000f60 tx=0x7fb030009040 comp rx=0 tx=0).stop 2024-04-25T01:22:36.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 -- 172.21.15.77:0/995384487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb040143d20 msgr2=0x7fb04015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb040143d20 0x7fb04015b8f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb03800c780 tx=0x7fb03800cc40 comp rx=0 tx=0).stop 2024-04-25T01:22:36.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 -- 172.21.15.77:0/995384487 shutdown_connections 2024-04-25T01:22:36.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb02006be30 0x7fb02006e2f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb040143d20 0x7fb04015b8f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb040143940 0x7fb04015b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 --2- 172.21.15.77:0/995384487 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb04006ed60 0x7fb04015dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 -- 172.21.15.77:0/995384487 >> 172.21.15.77:0/995384487 conn(0x7fb0400b1820 msgr2=0x7fb040139800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:36.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 -- 172.21.15.77:0/995384487 shutdown_connections 2024-04-25T01:22:36.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:36.692+0000 7fb044c15640 1 -- 172.21.15.77:0/995384487 wait complete. 2024-04-25T01:22:36.684 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T01:22:36.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:36.685 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-04-25T01:22:36.708 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:22:36.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:36.708 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T01:22:36.763 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:22:36.763 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:36.782 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:22:36.782 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:36.840 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:22:36.840 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:36.899 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:36.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.013+0000 7f20eb3a9640 1 Processor -- start 2024-04-25T01:22:36.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.014+0000 7f20eb3a9640 1 -- start start 2024-04-25T01:22:36.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.014+0000 7f20eb3a9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20e415e4c0 0x7f20e40d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.014+0000 7f20eb3a9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f20e40d2d40 0x7f20e40cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:36.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.014+0000 7f20eb3a9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e40d1e90 con 0x7f20e40d4a10 2024-04-25T01:22:36.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.014+0000 7f20eb3a9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f20e40b54a0 con 0x7f20e415e4c0 2024-04-25T01:22:36.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.014+0000 7f20eb3a9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f20e40d2130 con 0x7f20e40d2d40 2024-04-25T01:22:36.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.014+0000 7f20e9145640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20e415e4c0 0x7f20e40d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.014+0000 7f20e9145640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20e415e4c0 0x7f20e40d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47884/0 (socket says 172.21.15.77:47884) 2024-04-25T01:22:36.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.015+0000 7f20e9145640 1 -- 172.21.15.77:0/2310935939 learned_addr learned my addr 172.21.15.77:0/2310935939 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:36.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.015+0000 7f20e9946640 1 --2- 172.21.15.77:0/2310935939 >> v2:172.21.15.121:3300/0 conn(0x7f20e40d2d40 0x7f20e40cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:36.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.015+0000 7f20e9145640 1 -- 172.21.15.77:0/2310935939 >> v2:172.21.15.121:3300/0 conn(0x7f20e40d2d40 msgr2=0x7f20e40cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:36.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.015+0000 7f20e9145640 1 --2- 172.21.15.77:0/2310935939 >> v2:172.21.15.121:3300/0 conn(0x7f20e40d2d40 0x7f20e40cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.015+0000 7f20e9145640 1 -- 172.21.15.77:0/2310935939 >> v1:172.21.15.121:6789/0 conn(0x7f20e40d4a10 legacy=0x7f20e40b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:36.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.015+0000 7f20e9145640 1 -- 172.21.15.77:0/2310935939 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e40b5780 con 0x7f20e415e4c0 2024-04-25T01:22:36.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.015+0000 7f20e9145640 1 --2- 172.21.15.77:0/2310935939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e415e4c0 0x7f20e40d2800 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f20e408d2e0 tx=0x7f20e002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4ac26ccf1d5c8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:36.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.016+0000 7f20dbfff640 1 -- 172.21.15.77:0/2310935939 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0036070 con 0x7f20e415e4c0 2024-04-25T01:22:36.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.016+0000 7f20dbfff640 1 -- 172.21.15.77:0/2310935939 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e0031600 con 0x7f20e415e4c0 2024-04-25T01:22:36.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.016+0000 7f20dbfff640 1 -- 172.21.15.77:0/2310935939 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e00319d0 con 0x7f20e415e4c0 2024-04-25T01:22:36.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.016+0000 7f20eb3a9640 1 -- 172.21.15.77:0/2310935939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e415e4c0 msgr2=0x7f20e40d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:36.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.016+0000 7f20eb3a9640 1 --2- 172.21.15.77:0/2310935939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e415e4c0 0x7f20e40d2800 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f20e408d2e0 tx=0x7f20e002a660 comp rx=0 tx=0).stop 2024-04-25T01:22:36.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.016+0000 7f20eb3a9640 1 -- 172.21.15.77:0/2310935939 shutdown_connections 2024-04-25T01:22:36.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.016+0000 7f20eb3a9640 1 --2- 172.21.15.77:0/2310935939 >> v2:172.21.15.121:3300/0 conn(0x7f20e40d2d40 0x7f20e40cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.016+0000 7f20eb3a9640 1 --2- 172.21.15.77:0/2310935939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e415e4c0 0x7f20e40d2800 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:36.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.016+0000 7f20eb3a9640 1 -- 172.21.15.77:0/2310935939 >> 172.21.15.77:0/2310935939 conn(0x7f20e40d3ef0 msgr2=0x7f20e40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:36.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.016+0000 7f20eb3a9640 1 -- 172.21.15.77:0/2310935939 shutdown_connections 2024-04-25T01:22:36.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.016+0000 7f20eb3a9640 1 -- 172.21.15.77:0/2310935939 wait complete. 2024-04-25T01:22:37.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.017+0000 7f20eb3a9640 1 Processor -- start 2024-04-25T01:22:37.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.017+0000 7f20eb3a9640 1 -- start start 2024-04-25T01:22:37.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20eb3a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e40d2d40 0x7f20e4156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20eb3a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e40d4a10 0x7f20e41f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20eb3a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20e415e4c0 0x7f20e41f4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20eb3a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f20e415ea20 con 0x7f20e40d2d40 2024-04-25T01:22:37.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20eb3a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f20e415e8a0 con 0x7f20e40d4a10 2024-04-25T01:22:37.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20eb3a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f20e40b45f0 con 0x7f20e415e4c0 2024-04-25T01:22:37.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20e9145640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e40d2d40 0x7f20e4156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20e9145640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e40d2d40 0x7f20e4156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47894/0 (socket says 172.21.15.77:47894) 2024-04-25T01:22:37.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20e9145640 1 -- 172.21.15.77:0/986179255 learned_addr learned my addr 172.21.15.77:0/986179255 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:37.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20e8944640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e40d4a10 0x7f20e41f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20e9946640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20e415e4c0 0x7f20e41f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20e9145640 1 -- 172.21.15.77:0/986179255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20e415e4c0 msgr2=0x7f20e41f4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20e9145640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20e415e4c0 0x7f20e41f4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20e9145640 1 -- 172.21.15.77:0/986179255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e40d4a10 msgr2=0x7f20e41f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20e9145640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e40d4a10 0x7f20e41f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.018+0000 7f20e9145640 1 -- 172.21.15.77:0/986179255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e41f4e30 con 0x7f20e40d2d40 2024-04-25T01:22:37.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.019+0000 7f20e9145640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e40d2d40 0x7f20e4156290 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f20e0000f80 tx=0x7f20e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.019+0000 7f20d9ffb640 1 -- 172.21.15.77:0/986179255 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0036050 con 0x7f20e40d2d40 2024-04-25T01:22:37.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.019+0000 7f20d9ffb640 1 -- 172.21.15.77:0/986179255 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e0042070 con 0x7f20e40d2d40 2024-04-25T01:22:37.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.019+0000 7f20eb3a9640 1 -- 172.21.15.77:0/986179255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e41f5110 con 0x7f20e40d2d40 2024-04-25T01:22:37.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.019+0000 7f20d9ffb640 1 -- 172.21.15.77:0/986179255 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0009510 con 0x7f20e40d2d40 2024-04-25T01:22:37.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.019+0000 7f20eb3a9640 1 -- 172.21.15.77:0/986179255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e4156b10 con 0x7f20e40d2d40 2024-04-25T01:22:37.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.020+0000 7f20eb3a9640 1 -- 172.21.15.77:0/986179255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e4068fb0 con 0x7f20e40d2d40 2024-04-25T01:22:37.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.020+0000 7f20d9ffb640 1 -- 172.21.15.77:0/986179255 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f20e0007c70 con 0x7f20e40d2d40 2024-04-25T01:22:37.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.021+0000 7f20d9ffb640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f20c806be60 0x7f20c806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.021+0000 7f20d9ffb640 1 -- 172.21.15.77:0/986179255 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f20e00ac190 con 0x7f20e40d2d40 2024-04-25T01:22:37.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.021+0000 7f20e8944640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f20c806be60 0x7f20c806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.021+0000 7f20e8944640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f20c806be60 0x7f20c806e320 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f20e40b0cf0 tx=0x7f20d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.025+0000 7f20d9ffb640 1 -- 172.21.15.77:0/986179255 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f20e007a7c0 con 0x7f20e40d2d40 2024-04-25T01:22:37.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.199+0000 7f20eb3a9640 1 -- 172.21.15.77:0/986179255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20e40d38e0 con 0x7f20e40d2d40 2024-04-25T01:22:37.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.200+0000 7f20d9ffb640 1 -- 172.21.15.77:0/986179255 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18922 (secure 0 0 0) 0x7f20e007a0c0 con 0x7f20e40d2d40 2024-04-25T01:22:37.181 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:37.182 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:26.287128+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:37.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 -- 172.21.15.77:0/986179255 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f20c806be60 msgr2=0x7f20c806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f20c806be60 0x7f20c806e320 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f20e40b0cf0 tx=0x7f20d4009040 comp rx=0 tx=0).stop 2024-04-25T01:22:37.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 -- 172.21.15.77:0/986179255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e40d2d40 msgr2=0x7f20e4156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e40d2d40 0x7f20e4156290 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f20e0000f80 tx=0x7f20e0004a40 comp rx=0 tx=0).stop 2024-04-25T01:22:37.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 -- 172.21.15.77:0/986179255 shutdown_connections 2024-04-25T01:22:37.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f20c806be60 0x7f20c806e320 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20e415e4c0 0x7f20e41f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e40d4a10 0x7f20e41f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 --2- 172.21.15.77:0/986179255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e40d2d40 0x7f20e4156290 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 -- 172.21.15.77:0/986179255 >> 172.21.15.77:0/986179255 conn(0x7f20e40d3ef0 msgr2=0x7f20e40d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:37.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 -- 172.21.15.77:0/986179255 shutdown_connections 2024-04-25T01:22:37.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.202+0000 7f20eb3a9640 1 -- 172.21.15.77:0/986179255 wait complete. 2024-04-25T01:22:37.195 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:37.196 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:37.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.313+0000 7fa8e62bd640 1 Processor -- start 2024-04-25T01:22:37.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.313+0000 7fa8e62bd640 1 -- start start 2024-04-25T01:22:37.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.313+0000 7fa8e62bd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa8e00b47d0 0x7fa8e015b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.313+0000 7fa8e62bd640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e015c210 con 0x7fa8e00b37c0 2024-04-25T01:22:37.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.313+0000 7fa8e62bd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e00b2ce0 con 0x7fa8e0158a70 2024-04-25T01:22:37.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.313+0000 7fa8e62bd640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e00b2fc0 con 0x7fa8e00b47d0 2024-04-25T01:22:37.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.314+0000 7fa8e485a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa8e00b37c0 0x7fa8e015ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:32936/0 (socket says 172.21.15.77:32936) 2024-04-25T01:22:37.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.314+0000 7fa8df7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa8e00b47d0 0x7fa8e015b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.314+0000 7fa8e485a640 1 -- 172.21.15.77:0/3466316665 learned_addr learned my addr 172.21.15.77:0/3466316665 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:37.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.314+0000 7fa8df7fe640 1 -- 172.21.15.77:0/3466316665 >> v1:172.21.15.121:6789/0 conn(0x7fa8e0158a70 legacy=0x7fa8e00b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:37.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.314+0000 7fa8df7fe640 1 -- 172.21.15.77:0/3466316665 >> v1:172.21.15.77:6789/0 conn(0x7fa8e00b37c0 legacy=0x7fa8e015ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.314+0000 7fa8deffd640 1 -- 172.21.15.77:0/3466316665 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3976073373 0 0) 0x7fa8e015c210 con 0x7fa8e00b37c0 2024-04-25T01:22:37.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.314+0000 7fa8df7fe640 1 -- 172.21.15.77:0/3466316665 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e00b32a0 con 0x7fa8e00b47d0 2024-04-25T01:22:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.315+0000 7fa8df7fe640 1 --2- 172.21.15.77:0/3466316665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e00b47d0 0x7fa8e015b520 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40097d0 tx=0x7fa8d402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd7f361d7f8e629 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.315+0000 7fa8deffd640 1 -- 172.21.15.77:0/3466316665 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4037070 con 0x7fa8e00b47d0 2024-04-25T01:22:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.315+0000 7fa8deffd640 1 -- 172.21.15.77:0/3466316665 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8d4032440 con 0x7fa8e00b47d0 2024-04-25T01:22:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.315+0000 7fa8deffd640 1 -- 172.21.15.77:0/3466316665 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4032810 con 0x7fa8e00b47d0 2024-04-25T01:22:37.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.315+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3466316665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e00b47d0 msgr2=0x7fa8e015b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.315+0000 7fa8e62bd640 1 --2- 172.21.15.77:0/3466316665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e00b47d0 0x7fa8e015b520 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40097d0 tx=0x7fa8d402a210 comp rx=0 tx=0).stop 2024-04-25T01:22:37.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.315+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3466316665 shutdown_connections 2024-04-25T01:22:37.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.315+0000 7fa8e62bd640 1 --2- 172.21.15.77:0/3466316665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e00b47d0 0x7fa8e015b520 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.315+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3466316665 >> 172.21.15.77:0/3466316665 conn(0x7fa8e006b720 msgr2=0x7fa8e00b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:37.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.315+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3466316665 shutdown_connections 2024-04-25T01:22:37.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.315+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3466316665 wait complete. 2024-04-25T01:22:37.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.316+0000 7fa8e62bd640 1 Processor -- start 2024-04-25T01:22:37.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.317+0000 7fa8e62bd640 1 -- start start 2024-04-25T01:22:37.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.317+0000 7fa8e62bd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8e00b37c0 0x7fa8e013dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.317+0000 7fa8e62bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e00b47d0 0x7fa8e0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.317+0000 7fa8e62bd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8e0158a70 0x7fa8e0157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.317+0000 7fa8e62bd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e0166020 con 0x7fa8e00b47d0 2024-04-25T01:22:37.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.317+0000 7fa8e62bd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e0165ea0 con 0x7fa8e00b37c0 2024-04-25T01:22:37.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.317+0000 7fa8e62bd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa8e01661a0 con 0x7fa8e0158a70 2024-04-25T01:22:37.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.317+0000 7fa8df7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e00b47d0 0x7fa8e0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.317+0000 7fa8df7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e00b47d0 0x7fa8e0156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47906/0 (socket says 172.21.15.77:47906) 2024-04-25T01:22:37.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.317+0000 7fa8df7fe640 1 -- 172.21.15.77:0/3049586103 learned_addr learned my addr 172.21.15.77:0/3049586103 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:37.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8dffff640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8e00b37c0 0x7fa8e013dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8e485a640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8e0158a70 0x7fa8e0157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8df7fe640 1 -- 172.21.15.77:0/3049586103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8e0158a70 msgr2=0x7fa8e0157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8df7fe640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8e0158a70 0x7fa8e0157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8df7fe640 1 -- 172.21.15.77:0/3049586103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8e00b37c0 msgr2=0x7fa8e013dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8df7fe640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8e00b37c0 0x7fa8e013dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8df7fe640 1 -- 172.21.15.77:0/3049586103 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e00cd6a0 con 0x7fa8e00b47d0 2024-04-25T01:22:37.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8df7fe640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e00b47d0 0x7fa8e0156ed0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40098f0 tx=0x7fa8d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8dcff9640 1 -- 172.21.15.77:0/3049586103 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4037040 con 0x7fa8e00b47d0 2024-04-25T01:22:37.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8dcff9640 1 -- 172.21.15.77:0/3049586103 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8d4004230 con 0x7fa8e00b47d0 2024-04-25T01:22:37.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8dcff9640 1 -- 172.21.15.77:0/3049586103 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d403c730 con 0x7fa8e00b47d0 2024-04-25T01:22:37.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3049586103 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e013c010 con 0x7fa8e00b47d0 2024-04-25T01:22:37.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.318+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3049586103 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e013c5d0 con 0x7fa8e00b47d0 2024-04-25T01:22:37.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.319+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3049586103 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8e0068fb0 con 0x7fa8e00b47d0 2024-04-25T01:22:37.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.320+0000 7fa8dcff9640 1 -- 172.21.15.77:0/3049586103 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa8d403a070 con 0x7fa8e00b47d0 2024-04-25T01:22:37.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.320+0000 7fa8dcff9640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8cc06be30 0x7fa8cc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.320+0000 7fa8dcff9640 1 -- 172.21.15.77:0/3049586103 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fa8d40abb50 con 0x7fa8e00b47d0 2024-04-25T01:22:37.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.320+0000 7fa8dffff640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8cc06be30 0x7fa8cc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.320+0000 7fa8dffff640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8cc06be30 0x7fa8cc06e2f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0006fd0 tx=0x7fa8d0009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.324+0000 7fa8dcff9640 1 -- 172.21.15.77:0/3049586103 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa8d4041020 con 0x7fa8e00b47d0 2024-04-25T01:22:37.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.495+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3049586103 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa8e0141700 con 0x7fa8e00b47d0 2024-04-25T01:22:37.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.496+0000 7fa8dcff9640 1 -- 172.21.15.77:0/3049586103 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fa8d4079bc0 con 0x7fa8e00b47d0 2024-04-25T01:22:37.477 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:37.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.497+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3049586103 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8cc06be30 msgr2=0x7fa8cc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.497+0000 7fa8e62bd640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8cc06be30 0x7fa8cc06e2f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0006fd0 tx=0x7fa8d0009060 comp rx=0 tx=0).stop 2024-04-25T01:22:37.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.497+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3049586103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e00b47d0 msgr2=0x7fa8e0156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.497+0000 7fa8e62bd640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e00b47d0 0x7fa8e0156ed0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40098f0 tx=0x7fa8d40046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.497+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3049586103 shutdown_connections 2024-04-25T01:22:37.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.497+0000 7fa8e62bd640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8cc06be30 0x7fa8cc06e2f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.497+0000 7fa8e62bd640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e00b47d0 0x7fa8e0156ed0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.497+0000 7fa8e62bd640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8e0158a70 0x7fa8e0157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.497+0000 7fa8e62bd640 1 --2- 172.21.15.77:0/3049586103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8e00b37c0 0x7fa8e013dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.497+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3049586103 >> 172.21.15.77:0/3049586103 conn(0x7fa8e006b720 msgr2=0x7fa8e0159190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:37.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.498+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3049586103 shutdown_connections 2024-04-25T01:22:37.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.498+0000 7fa8e62bd640 1 -- 172.21.15.77:0/3049586103 wait complete. 2024-04-25T01:22:37.490 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:37.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.608+0000 7fbd08029640 1 Processor -- start 2024-04-25T01:22:37.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.608+0000 7fbd08029640 1 -- start start 2024-04-25T01:22:37.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd08029640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd000b37c0 0x7fbd0015bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd08029640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd0015c210 con 0x7fbd00158a70 2024-04-25T01:22:37.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd08029640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd000b2ce0 con 0x7fbd000b47d0 2024-04-25T01:22:37.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd08029640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbd000b2fc0 con 0x7fbd000b37c0 2024-04-25T01:22:37.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd05dc5640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbd00158a70 0x7fbd000b40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:32950/0 (socket says 172.21.15.77:32950) 2024-04-25T01:22:37.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd05dc5640 1 -- 172.21.15.77:0/914689989 learned_addr learned my addr 172.21.15.77:0/914689989 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:37.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 609674841 0 0) 0x7fbd0015c210 con 0x7fbd00158a70 2024-04-25T01:22:37.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcec003520 con 0x7fbd00158a70 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd065c6640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd000b37c0 0x7fbd0015bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1895629842 0 0) 0x7fbcec003520 con 0x7fbd00158a70 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd000b37c0 msgr2=0x7fbd0015bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd000b37c0 0x7fbd0015bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 >> v1:172.21.15.121:6789/0 conn(0x7fbd000b47d0 legacy=0x7fbd0015b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd000b32a0 con 0x7fbd00158a70 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fbcfc001670 con 0x7fbd00158a70 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 >> v1:172.21.15.77:6789/0 conn(0x7fbd00158a70 legacy=0x7fbd000b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcec0039e0 0x7fbcec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd000b37c0 0x7fbcec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec004d40 0x7fbcec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbd000b2fc0 con 0x7fbcec004d40 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec00baa0 con 0x7fbcec0039e0 2024-04-25T01:22:37.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.609+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbcec00deb0 con 0x7fbd000b37c0 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.610+0000 7fbd065c6640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec004d40 0x7fbcec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.610+0000 7fbd05dc5640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcec0039e0 0x7fbcec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.610+0000 7fbd055c4640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd000b37c0 0x7fbcec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.610+0000 7fbd065c6640 1 -- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd000b37c0 msgr2=0x7fbcec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.610+0000 7fbd065c6640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd000b37c0 0x7fbcec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.610+0000 7fbd065c6640 1 -- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcec0039e0 msgr2=0x7fbcec003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.610+0000 7fbd065c6640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcec0039e0 0x7fbcec003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.610+0000 7fbd065c6640 1 -- 172.21.15.77:0/914689989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcec003520 con 0x7fbcec004d40 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.610+0000 7fbd05dc5640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcec0039e0 0x7fbcec003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.610+0000 7fbd065c6640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec004d40 0x7fbcec0051c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009f90 tx=0x7fbcec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7f3a8b4dbe15bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.611+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8009c10 con 0x7fbcec004d40 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.611+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcf8036070 con 0x7fbcec004d40 2024-04-25T01:22:37.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.611+0000 7fbd04dc3640 1 -- 172.21.15.77:0/914689989 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8031850 con 0x7fbcec004d40 2024-04-25T01:22:37.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.611+0000 7fbd08029640 1 -- 172.21.15.77:0/914689989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec004d40 msgr2=0x7fbcec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.611+0000 7fbd08029640 1 --2- 172.21.15.77:0/914689989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec004d40 0x7fbcec0051c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009f90 tx=0x7fbcec005830 comp rx=0 tx=0).stop 2024-04-25T01:22:37.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.611+0000 7fbd065c6640 1 -- 172.21.15.77:0/914689989 reap_dead start 2024-04-25T01:22:37.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.611+0000 7fbd08029640 1 -- 172.21.15.77:0/914689989 shutdown_connections 2024-04-25T01:22:37.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.611+0000 7fbd08029640 1 -- 172.21.15.77:0/914689989 >> 172.21.15.77:0/914689989 conn(0x7fbd0006b720 msgr2=0x7fbd000b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:37.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.611+0000 7fbd08029640 1 -- 172.21.15.77:0/914689989 shutdown_connections 2024-04-25T01:22:37.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.611+0000 7fbd08029640 1 -- 172.21.15.77:0/914689989 wait complete. 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.612+0000 7fbd08029640 1 Processor -- start 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.612+0000 7fbd08029640 1 -- start start 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd08029640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd001505c0 0x7fbd0015da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd08029640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd001509a0 0x7fbd00154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd08029640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd0015e100 0x7fbd000b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd08029640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0015bdb0 con 0x7fbd001505c0 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd08029640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0015bc30 con 0x7fbd0015e100 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd08029640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbd0015bf30 con 0x7fbd001509a0 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd05dc5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd001505c0 0x7fbd0015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd05dc5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd001505c0 0x7fbd0015da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47922/0 (socket says 172.21.15.77:47922) 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd05dc5640 1 -- 172.21.15.77:0/4003470594 learned_addr learned my addr 172.21.15.77:0/4003470594 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd065c6640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd0015e100 0x7fbd000b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd055c4640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd001509a0 0x7fbd00154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd05dc5640 1 -- 172.21.15.77:0/4003470594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd001509a0 msgr2=0x7fbd00154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd05dc5640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd001509a0 0x7fbd00154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd05dc5640 1 -- 172.21.15.77:0/4003470594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd0015e100 msgr2=0x7fbd000b08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd05dc5640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd0015e100 0x7fbd000b08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.613+0000 7fbd05dc5640 1 -- 172.21.15.77:0/4003470594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd00154120 con 0x7fbd001505c0 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.614+0000 7fbd05dc5640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd001505c0 0x7fbd0015da60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc005fd0 tx=0x7fbcfc00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.614+0000 7fbcf67fc640 1 -- 172.21.15.77:0/4003470594 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00c8d0 con 0x7fbd001505c0 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.614+0000 7fbcf67fc640 1 -- 172.21.15.77:0/4003470594 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcfc018070 con 0x7fbd001505c0 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.614+0000 7fbd08029640 1 -- 172.21.15.77:0/4003470594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd00155210 con 0x7fbd001505c0 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.614+0000 7fbcf67fc640 1 -- 172.21.15.77:0/4003470594 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00cc70 con 0x7fbd001505c0 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.614+0000 7fbd08029640 1 -- 172.21.15.77:0/4003470594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd00153e40 con 0x7fbd001505c0 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.615+0000 7fbd08029640 1 -- 172.21.15.77:0/4003470594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd00068fb0 con 0x7fbd001505c0 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.615+0000 7fbcf67fc640 1 -- 172.21.15.77:0/4003470594 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbcfc013420 con 0x7fbd001505c0 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.616+0000 7fbcf67fc640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbce406be60 0x7fbce406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.616+0000 7fbcf67fc640 1 -- 172.21.15.77:0/4003470594 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fbcfc08aeb0 con 0x7fbd001505c0 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.616+0000 7fbd055c4640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbce406be60 0x7fbce406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.616+0000 7fbd055c4640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbce406be60 0x7fbce406e320 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd00157d20 tx=0x7fbcf0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.620+0000 7fbcf67fc640 1 -- 172.21.15.77:0/4003470594 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbcfc022250 con 0x7fbd001505c0 2024-04-25T01:22:37.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.793+0000 7fbd08029640 1 -- 172.21.15.77:0/4003470594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbd000b3c70 con 0x7fbd001505c0 2024-04-25T01:22:37.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.793+0000 7fbcf67fc640 1 -- 172.21.15.77:0/4003470594 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fbcfc059450 con 0x7fbd001505c0 2024-04-25T01:22:37.775 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:37.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 -- 172.21.15.77:0/4003470594 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbce406be60 msgr2=0x7fbce406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbce406be60 0x7fbce406e320 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd00157d20 tx=0x7fbcf0009040 comp rx=0 tx=0).stop 2024-04-25T01:22:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 -- 172.21.15.77:0/4003470594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd001505c0 msgr2=0x7fbd0015da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd001505c0 0x7fbd0015da60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc005fd0 tx=0x7fbcfc00ec90 comp rx=0 tx=0).stop 2024-04-25T01:22:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 -- 172.21.15.77:0/4003470594 shutdown_connections 2024-04-25T01:22:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbce406be60 0x7fbce406e320 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd0015e100 0x7fbd000b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbd001509a0 0x7fbd00154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 --2- 172.21.15.77:0/4003470594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd001505c0 0x7fbd0015da60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 -- 172.21.15.77:0/4003470594 >> 172.21.15.77:0/4003470594 conn(0x7fbd0006b720 msgr2=0x7fbd0006d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:37.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 -- 172.21.15.77:0/4003470594 shutdown_connections 2024-04-25T01:22:37.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.795+0000 7fbd08029640 1 -- 172.21.15.77:0/4003470594 wait complete. 2024-04-25T01:22:37.788 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:37.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.906+0000 7fb463f42640 1 Processor -- start 2024-04-25T01:22:37.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.907+0000 7fb463f42640 1 -- start start 2024-04-25T01:22:37.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.907+0000 7fb463f42640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb45c0720c0 0x7fb45c0b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.907+0000 7fb463f42640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c06caf0 0x7fb45c06d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.907+0000 7fb463f42640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb45c15bb80 con 0x7fb45c15b6a0 2024-04-25T01:22:37.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.907+0000 7fb463f42640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb45c0724a0 con 0x7fb45c0720c0 2024-04-25T01:22:37.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.907+0000 7fb463f42640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb45c0b3750 con 0x7fb45c06caf0 2024-04-25T01:22:37.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.907+0000 7fb4624df640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb45c15b6a0 0x7fb45c070b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:32954/0 (socket says 172.21.15.77:32954) 2024-04-25T01:22:37.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.907+0000 7fb4624df640 1 -- 172.21.15.77:0/1285124217 learned_addr learned my addr 172.21.15.77:0/1285124217 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:37.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3654482331 0 0) 0x7fb45c15bb80 con 0x7fb45c15b6a0 2024-04-25T01:22:37.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb4614dd640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c06caf0 0x7fb45c06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb448003520 con 0x7fb45c15b6a0 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb461cde640 1 --2- 172.21.15.77:0/1285124217 >> v2:172.21.15.121:3300/0 conn(0x7fb45c0720c0 0x7fb45c0b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2474871555 0 0) 0x7fb448003520 con 0x7fb45c15b6a0 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c06caf0 msgr2=0x7fb45c06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c06caf0 0x7fb45c06d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 >> v2:172.21.15.121:3300/0 conn(0x7fb45c0720c0 msgr2=0x7fb45c0b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 --2- 172.21.15.77:0/1285124217 >> v2:172.21.15.121:3300/0 conn(0x7fb45c0720c0 0x7fb45c0b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45c0b3a30 con 0x7fb45c15b6a0 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fb4540019f0 con 0x7fb45c15b6a0 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb461cde640 1 --2- 172.21.15.77:0/1285124217 >> v2:172.21.15.121:3300/0 conn(0x7fb45c0720c0 0x7fb45c0b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 >> v1:172.21.15.77:6789/0 conn(0x7fb45c15b6a0 legacy=0x7fb45c070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c06caf0 0x7fb4480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb448003f80 0x7fb448004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb448004d40 0x7fb4480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c0724a0 con 0x7fb448004d40 2024-04-25T01:22:37.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c0b3750 con 0x7fb448003f80 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb44800bb20 con 0x7fb45c06caf0 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb4624df640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb448004d40 0x7fb4480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb461cde640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c06caf0 0x7fb4480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb4614dd640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb448003f80 0x7fb448004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb4624df640 1 -- 172.21.15.77:0/1285124217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c06caf0 msgr2=0x7fb4480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb4624df640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c06caf0 0x7fb4480039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb4624df640 1 -- 172.21.15.77:0/1285124217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb448003f80 msgr2=0x7fb448004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb4624df640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb448003f80 0x7fb448004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.908+0000 7fb4624df640 1 -- 172.21.15.77:0/1285124217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb448003520 con 0x7fb448004d40 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.909+0000 7fb4624df640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb448004d40 0x7fb4480051c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb45c15bab0 tx=0x7fb448005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba366057a72a8de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.909+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45400a3d0 con 0x7fb448004d40 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.909+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb45402f070 con 0x7fb448004d40 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.909+0000 7fb460cdc640 1 -- 172.21.15.77:0/1285124217 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45402bbe0 con 0x7fb448004d40 2024-04-25T01:22:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.910+0000 7fb463f42640 1 -- 172.21.15.77:0/1285124217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb448004d40 msgr2=0x7fb4480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.910+0000 7fb463f42640 1 --2- 172.21.15.77:0/1285124217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb448004d40 0x7fb4480051c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb45c15bab0 tx=0x7fb448005830 comp rx=0 tx=0).stop 2024-04-25T01:22:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.910+0000 7fb4624df640 1 -- 172.21.15.77:0/1285124217 reap_dead start 2024-04-25T01:22:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.910+0000 7fb463f42640 1 -- 172.21.15.77:0/1285124217 shutdown_connections 2024-04-25T01:22:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.910+0000 7fb463f42640 1 -- 172.21.15.77:0/1285124217 >> 172.21.15.77:0/1285124217 conn(0x7fb45c06b720 msgr2=0x7fb45c0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.910+0000 7fb463f42640 1 -- 172.21.15.77:0/1285124217 shutdown_connections 2024-04-25T01:22:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.910+0000 7fb463f42640 1 -- 172.21.15.77:0/1285124217 wait complete. 2024-04-25T01:22:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.911+0000 7fb463f42640 1 Processor -- start 2024-04-25T01:22:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.911+0000 7fb463f42640 1 -- start start 2024-04-25T01:22:37.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb463f42640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c150640 0x7fb45c157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb463f42640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c150a20 0x7fb45c159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb463f42640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb45c14b320 0x7fb45c1566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb463f42640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c0cf3c0 con 0x7fb45c150640 2024-04-25T01:22:37.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb463f42640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c0b3ff0 con 0x7fb45c14b320 2024-04-25T01:22:37.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb463f42640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb45c0cf540 con 0x7fb45c150a20 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb461cde640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c150640 0x7fb45c157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb461cde640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c150640 0x7fb45c157340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47942/0 (socket says 172.21.15.77:47942) 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb461cde640 1 -- 172.21.15.77:0/625603823 learned_addr learned my addr 172.21.15.77:0/625603823 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb461cde640 1 -- 172.21.15.77:0/625603823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c150a20 msgr2=0x7fb45c159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb461cde640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c150a20 0x7fb45c159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb461cde640 1 -- 172.21.15.77:0/625603823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb45c14b320 msgr2=0x7fb45c1566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb461cde640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb45c14b320 0x7fb45c1566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb461cde640 1 -- 172.21.15.77:0/625603823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45c157a40 con 0x7fb45c150640 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb461cde640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c150640 0x7fb45c157340 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb458011700 tx=0x7fb458011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb4527fc640 1 -- 172.21.15.77:0/625603823 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb458019070 con 0x7fb45c150640 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb4527fc640 1 -- 172.21.15.77:0/625603823 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb458004220 con 0x7fb45c150640 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb463f42640 1 -- 172.21.15.77:0/625603823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45c157d20 con 0x7fb45c150640 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb4527fc640 1 -- 172.21.15.77:0/625603823 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45800f040 con 0x7fb45c150640 2024-04-25T01:22:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.912+0000 7fb463f42640 1 -- 172.21.15.77:0/625603823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb45c159cc0 con 0x7fb45c150640 2024-04-25T01:22:37.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.913+0000 7fb463f42640 1 -- 172.21.15.77:0/625603823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb45c068fb0 con 0x7fb45c150640 2024-04-25T01:22:37.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.914+0000 7fb4527fc640 1 -- 172.21.15.77:0/625603823 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb45800f6e0 con 0x7fb45c150640 2024-04-25T01:22:37.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.914+0000 7fb4527fc640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb44006be30 0x7fb44006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.914+0000 7fb4614dd640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb44006be30 0x7fb44006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.914+0000 7fb4527fc640 1 -- 172.21.15.77:0/625603823 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fb45808b490 con 0x7fb45c150640 2024-04-25T01:22:37.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.915+0000 7fb4614dd640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb44006be30 0x7fb44006e2f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb45c1545a0 tx=0x7fb44c009320 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:37.918+0000 7fb4527fc640 1 -- 172.21.15.77:0/625603823 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb458059ac0 con 0x7fb45c150640 2024-04-25T01:22:38.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.089+0000 7fb463f42640 1 -- 172.21.15.77:0/625603823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb45c0b3400 con 0x7fb45c150640 2024-04-25T01:22:38.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.090+0000 7fb4527fc640 1 -- 172.21.15.77:0/625603823 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fb4580593c0 con 0x7fb45c150640 2024-04-25T01:22:38.072 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:38.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 -- 172.21.15.77:0/625603823 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb44006be30 msgr2=0x7fb44006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb44006be30 0x7fb44006e2f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb45c1545a0 tx=0x7fb44c009320 comp rx=0 tx=0).stop 2024-04-25T01:22:38.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 -- 172.21.15.77:0/625603823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c150640 msgr2=0x7fb45c157340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c150640 0x7fb45c157340 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb458011700 tx=0x7fb458011bc0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 -- 172.21.15.77:0/625603823 shutdown_connections 2024-04-25T01:22:38.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb44006be30 0x7fb44006e2f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb45c14b320 0x7fb45c1566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb45c150a20 0x7fb45c159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 --2- 172.21.15.77:0/625603823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c150640 0x7fb45c157340 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 -- 172.21.15.77:0/625603823 >> 172.21.15.77:0/625603823 conn(0x7fb45c06b720 msgr2=0x7fb45c0b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:38.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 -- 172.21.15.77:0/625603823 shutdown_connections 2024-04-25T01:22:38.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.092+0000 7fb463f42640 1 -- 172.21.15.77:0/625603823 wait complete. 2024-04-25T01:22:38.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:38.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.206+0000 7fc4b98a5640 1 Processor -- start 2024-04-25T01:22:38.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.206+0000 7fc4b98a5640 1 -- start start 2024-04-25T01:22:38.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b98a5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc4b4158a60 0x7fc4b406df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b98a5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4b415f4c0 0x7fc4b40b3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b98a5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4b406d140 con 0x7fc4b4158680 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b98a5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fc4b406d420 con 0x7fc4b415f4c0 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b98a5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc4b406d700 con 0x7fc4b4158a60 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4b415f4c0 0x7fc4b40b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b2ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc4b4158680 0x7fc4b406e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:32958/0 (socket says 172.21.15.77:32958) 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b2ffd640 1 -- 172.21.15.77:0/1469501343 learned_addr learned my addr 172.21.15.77:0/1469501343 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4abfff640 1 --2- 172.21.15.77:0/1469501343 >> v2:172.21.15.121:3300/0 conn(0x7fc4b4158a60 0x7fc4b406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b37fe640 1 -- 172.21.15.77:0/1469501343 >> v2:172.21.15.121:3300/0 conn(0x7fc4b4158a60 msgr2=0x7fc4b406df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b37fe640 1 --2- 172.21.15.77:0/1469501343 >> v2:172.21.15.121:3300/0 conn(0x7fc4b4158a60 0x7fc4b406df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b37fe640 1 -- 172.21.15.77:0/1469501343 >> v1:172.21.15.77:6789/0 conn(0x7fc4b4158680 legacy=0x7fc4b406e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b37fe640 1 -- 172.21.15.77:0/1469501343 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b406d9e0 con 0x7fc4b415f4c0 2024-04-25T01:22:38.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.207+0000 7fc4b27fc640 1 -- 172.21.15.77:0/1469501343 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 24625915 0 0) 0x7fc4b406d140 con 0x7fc4b4158680 2024-04-25T01:22:38.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.208+0000 7fc4b37fe640 1 --2- 172.21.15.77:0/1469501343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b415f4c0 0x7fc4b40b3550 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4070fd0 tx=0x7fc4ac02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8732aa1381af7a5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:38.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.208+0000 7fc4b27fc640 1 -- 172.21.15.77:0/1469501343 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac007740 con 0x7fc4b415f4c0 2024-04-25T01:22:38.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.208+0000 7fc4b27fc640 1 -- 172.21.15.77:0/1469501343 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4ac0078e0 con 0x7fc4b415f4c0 2024-04-25T01:22:38.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.208+0000 7fc4b27fc640 1 -- 172.21.15.77:0/1469501343 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac007cb0 con 0x7fc4b415f4c0 2024-04-25T01:22:38.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.209+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1469501343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b415f4c0 msgr2=0x7fc4b40b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.209+0000 7fc4b98a5640 1 --2- 172.21.15.77:0/1469501343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b415f4c0 0x7fc4b40b3550 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4070fd0 tx=0x7fc4ac02a5f0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.209+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1469501343 shutdown_connections 2024-04-25T01:22:38.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.209+0000 7fc4b98a5640 1 --2- 172.21.15.77:0/1469501343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b415f4c0 0x7fc4b40b3550 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.209+0000 7fc4b98a5640 1 --2- 172.21.15.77:0/1469501343 >> v2:172.21.15.121:3300/0 conn(0x7fc4b4158a60 0x7fc4b406df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.209+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1469501343 >> 172.21.15.77:0/1469501343 conn(0x7fc4b406b720 msgr2=0x7fc4b40b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:38.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.209+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1469501343 shutdown_connections 2024-04-25T01:22:38.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.209+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1469501343 wait complete. 2024-04-25T01:22:38.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.210+0000 7fc4b98a5640 1 Processor -- start 2024-04-25T01:22:38.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.210+0000 7fc4b98a5640 1 -- start start 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.210+0000 7fc4b98a5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4b4158680 0x7fc4b413f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.210+0000 7fc4b98a5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4b4158a60 0x7fc4b414da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b98a5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b415f4c0 0x7fc4b4150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b98a5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b40b3c10 con 0x7fc4b415f4c0 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b98a5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b40b3a90 con 0x7fc4b4158680 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b98a5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc4b40b3d90 con 0x7fc4b4158a60 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b415f4c0 0x7fc4b4150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b415f4c0 0x7fc4b4150040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47960/0 (socket says 172.21.15.77:47960) 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b37fe640 1 -- 172.21.15.77:0/1023776472 learned_addr learned my addr 172.21.15.77:0/1023776472 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b2ffd640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4b4158680 0x7fc4b413f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4abfff640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4b4158a60 0x7fc4b414da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b37fe640 1 -- 172.21.15.77:0/1023776472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4b4158a60 msgr2=0x7fc4b414da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b37fe640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4b4158a60 0x7fc4b414da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b37fe640 1 -- 172.21.15.77:0/1023776472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4b4158680 msgr2=0x7fc4b413f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b37fe640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4b4158680 0x7fc4b413f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b37fe640 1 -- 172.21.15.77:0/1023776472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b4151470 con 0x7fc4b415f4c0 2024-04-25T01:22:38.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4abfff640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4b4158a60 0x7fc4b414da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:38.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b37fe640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b415f4c0 0x7fc4b4150040 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009e80 tx=0x7fc4ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:38.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b0ff9640 1 -- 172.21.15.77:0/1023776472 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac03f070 con 0x7fc4b415f4c0 2024-04-25T01:22:38.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b0ff9640 1 -- 172.21.15.77:0/1023776472 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4ac031c40 con 0x7fc4b415f4c0 2024-04-25T01:22:38.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1023776472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b4150580 con 0x7fc4b415f4c0 2024-04-25T01:22:38.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b0ff9640 1 -- 172.21.15.77:0/1023776472 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac009450 con 0x7fc4b415f4c0 2024-04-25T01:22:38.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.211+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1023776472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b4150b00 con 0x7fc4b415f4c0 2024-04-25T01:22:38.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.212+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1023776472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4b40679d0 con 0x7fc4b415f4c0 2024-04-25T01:22:38.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.213+0000 7fc4b0ff9640 1 -- 172.21.15.77:0/1023776472 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc4ac003990 con 0x7fc4b415f4c0 2024-04-25T01:22:38.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.213+0000 7fc4b0ff9640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc49406be30 0x7fc49406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.213+0000 7fc4b0ff9640 1 -- 172.21.15.77:0/1023776472 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fc4ac0ac3c0 con 0x7fc4b415f4c0 2024-04-25T01:22:38.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.213+0000 7fc4b2ffd640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc49406be30 0x7fc49406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.213+0000 7fc4b2ffd640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc49406be30 0x7fc49406e2f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4000f60 tx=0x7fc4a4007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:38.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.217+0000 7fc4b0ff9640 1 -- 172.21.15.77:0/1023776472 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc4b40679d0 con 0x7fc4b415f4c0 2024-04-25T01:22:38.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.390+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1023776472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc4b4151640 con 0x7fc4b415f4c0 2024-04-25T01:22:38.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.391+0000 7fc4b0ff9640 1 -- 172.21.15.77:0/1023776472 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fc4ac07a9f0 con 0x7fc4b415f4c0 2024-04-25T01:22:38.372 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:38.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1023776472 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc49406be30 msgr2=0x7fc49406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc49406be30 0x7fc49406e2f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4000f60 tx=0x7fc4a4007c00 comp rx=0 tx=0).stop 2024-04-25T01:22:38.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1023776472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b415f4c0 msgr2=0x7fc4b4150040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b415f4c0 0x7fc4b4150040 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009e80 tx=0x7fc4ac004a40 comp rx=0 tx=0).stop 2024-04-25T01:22:38.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1023776472 shutdown_connections 2024-04-25T01:22:38.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc49406be30 0x7fc49406e2f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b415f4c0 0x7fc4b4150040 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4b4158a60 0x7fc4b414da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 --2- 172.21.15.77:0/1023776472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4b4158680 0x7fc4b413f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1023776472 >> 172.21.15.77:0/1023776472 conn(0x7fc4b406b720 msgr2=0x7fc4b40af670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:38.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1023776472 shutdown_connections 2024-04-25T01:22:38.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.393+0000 7fc4b98a5640 1 -- 172.21.15.77:0/1023776472 wait complete. 2024-04-25T01:22:38.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:22:38.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.500+0000 7ff0a7b90640 1 Processor -- start 2024-04-25T01:22:38.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.501+0000 7ff0a7b90640 1 -- start start 2024-04-25T01:22:38.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.501+0000 7ff0a7b90640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0a0143940 0x7ff0a006e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.501+0000 7ff0a7b90640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0a0070e60 con 0x7ff0a015f400 2024-04-25T01:22:38.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.501+0000 7ff0a7b90640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0a0071140 con 0x7ff0a0143d20 2024-04-25T01:22:38.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.501+0000 7ff0a7b90640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff0a0071420 con 0x7ff0a0143940 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.501+0000 7ff0a612d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff0a015f400 0x7ff0a0071c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:32970/0 (socket says 172.21.15.77:32970) 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.501+0000 7ff0a612d640 1 -- 172.21.15.77:0/1780365849 learned_addr learned my addr 172.21.15.77:0/1780365849 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a592c640 1 --2- 172.21.15.77:0/1780365849 >> v2:172.21.15.121:3300/0 conn(0x7ff0a0143940 0x7ff0a006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1353572825 0 0) 0x7ff0a0070e60 con 0x7ff0a015f400 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff08c005730 con 0x7ff0a015f400 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 241882710 0 0) 0x7ff08c005730 con 0x7ff0a015f400 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 >> v2:172.21.15.121:3300/0 conn(0x7ff0a0143940 msgr2=0x7ff0a006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 --2- 172.21.15.77:0/1780365849 >> v2:172.21.15.121:3300/0 conn(0x7ff0a0143940 0x7ff0a006e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 >> v1:172.21.15.121:6789/0 conn(0x7ff0a0143d20 legacy=0x7ff0a015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a0071700 con 0x7ff0a015f400 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7ff098001670 con 0x7ff0a015f400 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 >> v1:172.21.15.77:6789/0 conn(0x7ff0a015f400 legacy=0x7ff0a0071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 0x7ff08c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff08c006a10 0x7ff08c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff08c0073d0 0x7ff08c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a0071420 con 0x7ff08c005bc0 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c00e270 con 0x7ff08c006a10 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff08c00e470 con 0x7ff08c0073d0 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a592c640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 0x7ff08c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a512b640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff08c006a10 0x7ff08c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a612d640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff08c0073d0 0x7ff08c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a592c640 1 -- 172.21.15.77:0/1780365849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff08c0073d0 msgr2=0x7ff08c00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a592c640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff08c0073d0 0x7ff08c00dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a592c640 1 -- 172.21.15.77:0/1780365849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff08c006a10 msgr2=0x7ff08c006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a592c640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff08c006a10 0x7ff08c006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a592c640 1 -- 172.21.15.77:0/1780365849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08c005730 con 0x7ff08c005bc0 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a512b640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff08c006a10 0x7ff08c006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.502+0000 7ff0a612d640 1 -- 172.21.15.77:0/1780365849 reap_dead start 2024-04-25T01:22:38.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.503+0000 7ff0a592c640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 0x7ff08c005fe0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff09c009fc0 tx=0x7ff08c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f810237ff6ea456 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:38.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.503+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c009c10 con 0x7ff08c005bc0 2024-04-25T01:22:38.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.503+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff09c036070 con 0x7ff08c005bc0 2024-04-25T01:22:38.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.503+0000 7ff0a492a640 1 -- 172.21.15.77:0/1780365849 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c031880 con 0x7ff08c005bc0 2024-04-25T01:22:38.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.504+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1780365849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 msgr2=0x7ff08c005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.504+0000 7ff0a7b90640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 0x7ff08c005fe0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff09c009fc0 tx=0x7ff08c007a40 comp rx=0 tx=0).stop 2024-04-25T01:22:38.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.504+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1780365849 shutdown_connections 2024-04-25T01:22:38.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.504+0000 7ff0a7b90640 1 --2- 172.21.15.77:0/1780365849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 0x7ff08c005fe0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.504+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1780365849 >> 172.21.15.77:0/1780365849 conn(0x7ff0a00b1820 msgr2=0x7ff0a00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:38.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.504+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1780365849 shutdown_connections 2024-04-25T01:22:38.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.504+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1780365849 wait complete. 2024-04-25T01:22:38.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.505+0000 7ff0a7b90640 1 Processor -- start 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.505+0000 7ff0a7b90640 1 -- start start 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a7b90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 0x7ff0a01ef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a7b90640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a0159950 0x7ff0a0159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a7b90640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0a015d3f0 0x7ff0a015d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a7b90640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a015f080 con 0x7ff08c005bc0 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a7b90640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a015ef00 con 0x7ff0a015d3f0 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a7b90640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a015f200 con 0x7ff0a0159950 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a592c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 0x7ff0a01ef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a592c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 0x7ff0a01ef0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47984/0 (socket says 172.21.15.77:47984) 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a592c640 1 -- 172.21.15.77:0/1389465595 learned_addr learned my addr 172.21.15.77:0/1389465595 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a592c640 1 -- 172.21.15.77:0/1389465595 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a0159950 msgr2=0x7ff0a0159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a592c640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a0159950 0x7ff0a0159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a592c640 1 -- 172.21.15.77:0/1389465595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0a015d3f0 msgr2=0x7ff0a015d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a592c640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0a015d3f0 0x7ff0a015d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a592c640 1 -- 172.21.15.77:0/1389465595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a00ccff0 con 0x7ff08c005bc0 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a592c640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 0x7ff0a01ef0e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff09c02aee0 tx=0x7ff09c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0967fc640 1 -- 172.21.15.77:0/1389465595 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c009c10 con 0x7ff08c005bc0 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0967fc640 1 -- 172.21.15.77:0/1389465595 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff09c003990 con 0x7ff08c005bc0 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1389465595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a015a410 con 0x7ff08c005bc0 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0967fc640 1 -- 172.21.15.77:0/1389465595 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c003d60 con 0x7ff08c005bc0 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.506+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1389465595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a015a9d0 con 0x7ff08c005bc0 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.507+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1389465595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0a0068fb0 con 0x7ff08c005bc0 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.508+0000 7ff0967fc640 1 -- 172.21.15.77:0/1389465595 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff09c005d10 con 0x7ff08c005bc0 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.508+0000 7ff0967fc640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff08406be30 0x7ff08406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.508+0000 7ff0a512b640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff08406be30 0x7ff08406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.508+0000 7ff0967fc640 1 -- 172.21.15.77:0/1389465595 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7ff09c0ac5c0 con 0x7ff08c005bc0 2024-04-25T01:22:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.509+0000 7ff0a512b640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff08406be30 0x7ff08406e2f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff0a006ea00 tx=0x7ff090009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:38.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.512+0000 7ff0967fc640 1 -- 172.21.15.77:0/1389465595 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff09c07ab60 con 0x7ff08c005bc0 2024-04-25T01:22:38.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.695+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1389465595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff0a006d150 con 0x7ff08c005bc0 2024-04-25T01:22:38.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.695+0000 7ff0967fc640 1 -- 172.21.15.77:0/1389465595 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7ff09c07a460 con 0x7ff08c005bc0 2024-04-25T01:22:38.677 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1389465595 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff08406be30 msgr2=0x7ff08406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff08406be30 0x7ff08406e2f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff0a006ea00 tx=0x7ff090009040 comp rx=0 tx=0).stop 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1389465595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 msgr2=0x7ff0a01ef0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 0x7ff0a01ef0e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff09c02aee0 tx=0x7ff09c009750 comp rx=0 tx=0).stop 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1389465595 shutdown_connections 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff08406be30 0x7ff08406e2f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0a015d3f0 0x7ff0a015d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a0159950 0x7ff0a0159dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 --2- 172.21.15.77:0/1389465595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff08c005bc0 0x7ff0a01ef0e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1389465595 >> 172.21.15.77:0/1389465595 conn(0x7ff0a00b1820 msgr2=0x7ff0a00b00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1389465595 shutdown_connections 2024-04-25T01:22:38.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:38.697+0000 7ff0a7b90640 1 -- 172.21.15.77:0/1389465595 wait complete. 2024-04-25T01:22:38.689 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T01:22:38.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:38.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-04-25T01:22:38.720 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:22:38.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:38.720 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T01:22:38.783 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:22:38.783 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:38.801 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:22:38.801 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:38.856 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:22:38.856 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:38.911 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:39.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.066+0000 7fc875a9a640 1 Processor -- start 2024-04-25T01:22:39.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.067+0000 7fc875a9a640 1 -- start start 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.067+0000 7fc875a9a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc870143940 0x7fc87006e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.067+0000 7fc875a9a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc870070e60 con 0x7fc87015f400 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.067+0000 7fc875a9a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc870071140 con 0x7fc870143d20 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.067+0000 7fc875a9a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc870071420 con 0x7fc870143940 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.067+0000 7fc86ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc87015f400 0x7fc870071c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:32986/0 (socket says 172.21.15.77:32986) 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86ffff640 1 -- 172.21.15.77:0/1557545468 learned_addr learned my addr 172.21.15.77:0/1557545468 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2303553091 0 0) 0x7fc870070e60 con 0x7fc87015f400 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86f7fe640 1 --2- 172.21.15.77:0/1557545468 >> v2:172.21.15.121:3300/0 conn(0x7fc870143940 0x7fc87006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc858003520 con 0x7fc87015f400 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2347104318 0 0) 0x7fc858003520 con 0x7fc87015f400 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 >> v2:172.21.15.121:3300/0 conn(0x7fc870143940 msgr2=0x7fc87006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 --2- 172.21.15.77:0/1557545468 >> v2:172.21.15.121:3300/0 conn(0x7fc870143940 0x7fc87006e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 >> v1:172.21.15.121:6789/0 conn(0x7fc870143d20 legacy=0x7fc87015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc870071700 con 0x7fc87015f400 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fc8680019f0 con 0x7fc87015f400 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 >> v1:172.21.15.77:6789/0 conn(0x7fc87015f400 legacy=0x7fc870071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8580039b0 0x7fc858003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc858004800 0x7fc858004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8580051c0 0x7fc85800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc870071420 con 0x7fc8580051c0 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc85800c060 con 0x7fc858004800 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc85800c260 con 0x7fc8580039b0 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86ffff640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8580051c0 0x7fc85800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86f7fe640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8580039b0 0x7fc858003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86effd640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc858004800 0x7fc858004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86ffff640 1 -- 172.21.15.77:0/1557545468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8580039b0 msgr2=0x7fc858003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86ffff640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8580039b0 0x7fc858003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86ffff640 1 -- 172.21.15.77:0/1557545468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc858004800 msgr2=0x7fc858004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86ffff640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc858004800 0x7fc858004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86ffff640 1 -- 172.21.15.77:0/1557545468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc858003520 con 0x7fc8580051c0 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.068+0000 7fc86effd640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc858004800 0x7fc858004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.069+0000 7fc86ffff640 1 -- 172.21.15.77:0/1557545468 reap_dead start 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.069+0000 7fc86ffff640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8580051c0 0x7fc85800b9c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc86800a800 tx=0x7fc858005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b47deede1da56c80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.069+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86800a450 con 0x7fc8580051c0 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.069+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc86802f070 con 0x7fc8580051c0 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.069+0000 7fc86e7fc640 1 -- 172.21.15.77:0/1557545468 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86802bc90 con 0x7fc8580051c0 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.070+0000 7fc875a9a640 1 -- 172.21.15.77:0/1557545468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8580051c0 msgr2=0x7fc85800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.070+0000 7fc875a9a640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8580051c0 0x7fc85800b9c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc86800a800 tx=0x7fc858005830 comp rx=0 tx=0).stop 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.070+0000 7fc875a9a640 1 -- 172.21.15.77:0/1557545468 shutdown_connections 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.070+0000 7fc875a9a640 1 --2- 172.21.15.77:0/1557545468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8580051c0 0x7fc85800b9c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.070+0000 7fc875a9a640 1 -- 172.21.15.77:0/1557545468 >> 172.21.15.77:0/1557545468 conn(0x7fc8700b1820 msgr2=0x7fc8700b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:39.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.070+0000 7fc875a9a640 1 -- 172.21.15.77:0/1557545468 shutdown_connections 2024-04-25T01:22:39.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.070+0000 7fc875a9a640 1 -- 172.21.15.77:0/1557545468 wait complete. 2024-04-25T01:22:39.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.071+0000 7fc875a9a640 1 Processor -- start 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.071+0000 7fc875a9a640 1 -- start start 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc875a9a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8580051c0 0x7fc87015ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc875a9a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc870136ba0 0x7fc870141720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc875a9a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc870141c60 0x7fc8701f1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc875a9a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc87015f080 con 0x7fc870141c60 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc875a9a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc87015ef00 con 0x7fc8580051c0 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc875a9a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc87015f200 con 0x7fc870136ba0 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc870141c60 0x7fc8701f1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc870141c60 0x7fc8701f1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48002/0 (socket says 172.21.15.77:48002) 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86ffff640 1 -- 172.21.15.77:0/2445330307 learned_addr learned my addr 172.21.15.77:0/2445330307 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86effd640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc870136ba0 0x7fc870141720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86f7fe640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8580051c0 0x7fc87015ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86ffff640 1 -- 172.21.15.77:0/2445330307 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc870136ba0 msgr2=0x7fc870141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86ffff640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc870136ba0 0x7fc870141720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86ffff640 1 -- 172.21.15.77:0/2445330307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8580051c0 msgr2=0x7fc87015ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86ffff640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8580051c0 0x7fc87015ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86ffff640 1 -- 172.21.15.77:0/2445330307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8701f28d0 con 0x7fc870141c60 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86effd640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc870136ba0 0x7fc870141720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc86ffff640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc870141c60 0x7fc8701f1920 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc868002230 tx=0x7fc870071030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.072+0000 7fc857fff640 1 -- 172.21.15.77:0/2445330307 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86800a450 con 0x7fc870141c60 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.073+0000 7fc857fff640 1 -- 172.21.15.77:0/2445330307 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc868034470 con 0x7fc870141c60 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.073+0000 7fc875a9a640 1 -- 172.21.15.77:0/2445330307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8701f25f0 con 0x7fc870141c60 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.073+0000 7fc857fff640 1 -- 172.21.15.77:0/2445330307 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc868001f70 con 0x7fc870141c60 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.073+0000 7fc875a9a640 1 -- 172.21.15.77:0/2445330307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8701363a0 con 0x7fc870141c60 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.074+0000 7fc875a9a640 1 -- 172.21.15.77:0/2445330307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc870068fb0 con 0x7fc870141c60 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.074+0000 7fc857fff640 1 -- 172.21.15.77:0/2445330307 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc868034840 con 0x7fc870141c60 2024-04-25T01:22:39.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.074+0000 7fc857fff640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc85c06be60 0x7fc85c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.075+0000 7fc857fff640 1 -- 172.21.15.77:0/2445330307 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fc8680ab520 con 0x7fc870141c60 2024-04-25T01:22:39.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.075+0000 7fc86f7fe640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc85c06be60 0x7fc85c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.075+0000 7fc86f7fe640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc85c06be60 0x7fc85c06e320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc8700cc280 tx=0x7fc864008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.079+0000 7fc857fff640 1 -- 172.21.15.77:0/2445330307 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc868079b50 con 0x7fc870141c60 2024-04-25T01:22:39.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.254+0000 7fc875a9a640 1 -- 172.21.15.77:0/2445330307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc87006d150 con 0x7fc870141c60 2024-04-25T01:22:39.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.255+0000 7fc857fff640 1 -- 172.21.15.77:0/2445330307 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18922 (secure 0 0 0) 0x7fc868079450 con 0x7fc870141c60 2024-04-25T01:22:39.237 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:39.237 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:26.287128+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:39.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 -- 172.21.15.77:0/2445330307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc85c06be60 msgr2=0x7fc85c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc85c06be60 0x7fc85c06e320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc8700cc280 tx=0x7fc864008040 comp rx=0 tx=0).stop 2024-04-25T01:22:39.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 -- 172.21.15.77:0/2445330307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc870141c60 msgr2=0x7fc8701f1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc870141c60 0x7fc8701f1920 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc868002230 tx=0x7fc870071030 comp rx=0 tx=0).stop 2024-04-25T01:22:39.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 -- 172.21.15.77:0/2445330307 shutdown_connections 2024-04-25T01:22:39.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc85c06be60 0x7fc85c06e320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc870141c60 0x7fc8701f1920 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc870136ba0 0x7fc870141720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 --2- 172.21.15.77:0/2445330307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8580051c0 0x7fc87015ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 -- 172.21.15.77:0/2445330307 >> 172.21.15.77:0/2445330307 conn(0x7fc8700b1820 msgr2=0x7fc8700b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:39.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 -- 172.21.15.77:0/2445330307 shutdown_connections 2024-04-25T01:22:39.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.257+0000 7fc875a9a640 1 -- 172.21.15.77:0/2445330307 wait complete. 2024-04-25T01:22:39.250 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:39.250 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:39.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.369+0000 7fe57a3ce640 1 Processor -- start 2024-04-25T01:22:39.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.369+0000 7fe57a3ce640 1 -- start start 2024-04-25T01:22:39.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.370+0000 7fe57a3ce640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe5740b9a90 0x7fe5740d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.370+0000 7fe57a3ce640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe5740d27b0 0x7fe5740d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.370+0000 7fe57a3ce640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5740cea40 con 0x7fe5740d23d0 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.370+0000 7fe57a3ce640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fe5740cece0 con 0x7fe5740b9a90 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.370+0000 7fe57a3ce640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe5740cefc0 con 0x7fe5740d27b0 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.370+0000 7fe5737fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe5740b9a90 0x7fe5740d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.370+0000 7fe5737fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe5740b9a90 0x7fe5740d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48010/0 (socket says 172.21.15.77:48010) 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.370+0000 7fe5737fe640 1 -- 172.21.15.77:0/3449175444 learned_addr learned my addr 172.21.15.77:0/3449175444 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.371+0000 7fe57896b640 1 --2- 172.21.15.77:0/3449175444 >> v2:172.21.15.121:3300/0 conn(0x7fe5740d27b0 0x7fe5740d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.371+0000 7fe5737fe640 1 -- 172.21.15.77:0/3449175444 >> v2:172.21.15.121:3300/0 conn(0x7fe5740d27b0 msgr2=0x7fe5740d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.371+0000 7fe5737fe640 1 --2- 172.21.15.77:0/3449175444 >> v2:172.21.15.121:3300/0 conn(0x7fe5740d27b0 0x7fe5740d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.371+0000 7fe5737fe640 1 -- 172.21.15.77:0/3449175444 >> v1:172.21.15.121:6789/0 conn(0x7fe5740d23d0 legacy=0x7fe5740b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.371+0000 7fe5737fe640 1 -- 172.21.15.77:0/3449175444 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5740cf2a0 con 0x7fe5740b9a90 2024-04-25T01:22:39.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.371+0000 7fe57896b640 1 --2- 172.21.15.77:0/3449175444 >> v2:172.21.15.121:3300/0 conn(0x7fe5740d27b0 0x7fe5740d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.371+0000 7fe5737fe640 1 --2- 172.21.15.77:0/3449175444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5740b9a90 0x7fe5740d4920 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe574067900 tx=0x7fe56802a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3a591ca27a6aba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.371+0000 7fe572ffd640 1 -- 172.21.15.77:0/3449175444 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe568037070 con 0x7fe5740b9a90 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.371+0000 7fe572ffd640 1 -- 172.21.15.77:0/3449175444 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe568032440 con 0x7fe5740b9a90 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.371+0000 7fe572ffd640 1 -- 172.21.15.77:0/3449175444 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe568032810 con 0x7fe5740b9a90 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.372+0000 7fe57a3ce640 1 -- 172.21.15.77:0/3449175444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5740b9a90 msgr2=0x7fe5740d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.372+0000 7fe57a3ce640 1 --2- 172.21.15.77:0/3449175444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5740b9a90 0x7fe5740d4920 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe574067900 tx=0x7fe56802a290 comp rx=0 tx=0).stop 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.372+0000 7fe57a3ce640 1 -- 172.21.15.77:0/3449175444 shutdown_connections 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.372+0000 7fe57a3ce640 1 --2- 172.21.15.77:0/3449175444 >> v2:172.21.15.121:3300/0 conn(0x7fe5740d27b0 0x7fe5740d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.372+0000 7fe57a3ce640 1 --2- 172.21.15.77:0/3449175444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5740b9a90 0x7fe5740d4920 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.372+0000 7fe57a3ce640 1 -- 172.21.15.77:0/3449175444 >> 172.21.15.77:0/3449175444 conn(0x7fe5740d3e30 msgr2=0x7fe5740d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.372+0000 7fe57a3ce640 1 -- 172.21.15.77:0/3449175444 shutdown_connections 2024-04-25T01:22:39.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.372+0000 7fe57a3ce640 1 -- 172.21.15.77:0/3449175444 wait complete. 2024-04-25T01:22:39.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.373+0000 7fe57a3ce640 1 Processor -- start 2024-04-25T01:22:39.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.373+0000 7fe57a3ce640 1 -- start start 2024-04-25T01:22:39.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe57a3ce640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5740b9a90 0x7fe5741513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe57a3ce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5740d23d0 0x7fe5740d4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe57a3ce640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5740d27b0 0x7fe5740d4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe57a3ce640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe5740cfd80 con 0x7fe5740d23d0 2024-04-25T01:22:39.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe57a3ce640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe5740cf860 con 0x7fe5740b9a90 2024-04-25T01:22:39.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe57a3ce640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe5740b6c80 con 0x7fe5740d27b0 2024-04-25T01:22:39.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe5737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5740d23d0 0x7fe5740d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe5737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5740d23d0 0x7fe5740d4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48024/0 (socket says 172.21.15.77:48024) 2024-04-25T01:22:39.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe5737fe640 1 -- 172.21.15.77:0/723137113 learned_addr learned my addr 172.21.15.77:0/723137113 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe57896b640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5740d27b0 0x7fe5740d4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe573fff640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5740b9a90 0x7fe5741513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe5737fe640 1 -- 172.21.15.77:0/723137113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5740d27b0 msgr2=0x7fe5740d4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe5737fe640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5740d27b0 0x7fe5740d4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe5737fe640 1 -- 172.21.15.77:0/723137113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5740b9a90 msgr2=0x7fe5741513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe5737fe640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5740b9a90 0x7fe5741513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe5737fe640 1 -- 172.21.15.77:0/723137113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5741518e0 con 0x7fe5740d23d0 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe57896b640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5740d27b0 0x7fe5740d4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.374+0000 7fe573fff640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5740b9a90 0x7fe5741513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.375+0000 7fe5737fe640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5740d23d0 0x7fe5740d4210 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe568000c00 tx=0x7fe5680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.375+0000 7fe570ff9640 1 -- 172.21.15.77:0/723137113 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe568041020 con 0x7fe5740d23d0 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.375+0000 7fe570ff9640 1 -- 172.21.15.77:0/723137113 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe568030970 con 0x7fe5740d23d0 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.375+0000 7fe57a3ce640 1 -- 172.21.15.77:0/723137113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57409e880 con 0x7fe5740d23d0 2024-04-25T01:22:39.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.375+0000 7fe570ff9640 1 -- 172.21.15.77:0/723137113 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe568037040 con 0x7fe5740d23d0 2024-04-25T01:22:39.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.375+0000 7fe57a3ce640 1 -- 172.21.15.77:0/723137113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57409ee40 con 0x7fe5740d23d0 2024-04-25T01:22:39.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.376+0000 7fe57a3ce640 1 -- 172.21.15.77:0/723137113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5740c89b0 con 0x7fe5740d23d0 2024-04-25T01:22:39.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.376+0000 7fe570ff9640 1 -- 172.21.15.77:0/723137113 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe568044020 con 0x7fe5740d23d0 2024-04-25T01:22:39.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.377+0000 7fe570ff9640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56006be30 0x7fe56006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.377+0000 7fe570ff9640 1 -- 172.21.15.77:0/723137113 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fe5680ab940 con 0x7fe5740d23d0 2024-04-25T01:22:39.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.377+0000 7fe573fff640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56006be30 0x7fe56006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.377+0000 7fe573fff640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56006be30 0x7fe56006e2f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe564003eb0 tx=0x7fe564009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.381+0000 7fe570ff9640 1 -- 172.21.15.77:0/723137113 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe5740c89b0 con 0x7fe5740d23d0 2024-04-25T01:22:39.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.551+0000 7fe57a3ce640 1 -- 172.21.15.77:0/723137113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe574068fb0 con 0x7fe5740d23d0 2024-04-25T01:22:39.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.551+0000 7fe570ff9640 1 -- 172.21.15.77:0/723137113 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fe56804b070 con 0x7fe5740d23d0 2024-04-25T01:22:39.533 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:39.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 -- 172.21.15.77:0/723137113 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56006be30 msgr2=0x7fe56006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56006be30 0x7fe56006e2f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe564003eb0 tx=0x7fe564009040 comp rx=0 tx=0).stop 2024-04-25T01:22:39.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 -- 172.21.15.77:0/723137113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5740d23d0 msgr2=0x7fe5740d4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5740d23d0 0x7fe5740d4210 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe568000c00 tx=0x7fe5680046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 -- 172.21.15.77:0/723137113 shutdown_connections 2024-04-25T01:22:39.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56006be30 0x7fe56006e2f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5740d23d0 0x7fe5740d4210 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5740d27b0 0x7fe5740d4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 --2- 172.21.15.77:0/723137113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5740b9a90 0x7fe5741513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 -- 172.21.15.77:0/723137113 >> 172.21.15.77:0/723137113 conn(0x7fe5740d3e30 msgr2=0x7fe5740b8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:39.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 -- 172.21.15.77:0/723137113 shutdown_connections 2024-04-25T01:22:39.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.553+0000 7fe57a3ce640 1 -- 172.21.15.77:0/723137113 wait complete. 2024-04-25T01:22:39.546 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:39.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.664+0000 7f907b728640 1 Processor -- start 2024-04-25T01:22:39.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.664+0000 7f907b728640 1 -- start start 2024-04-25T01:22:39.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f907b728640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9074158a70 0x7f90740b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f907b728640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90740b37c0 0x7f90740b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f907b728640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f907415c210 con 0x7f907415b520 2024-04-25T01:22:39.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f907b728640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f90740b2ce0 con 0x7f9074158a70 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f907b728640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f90740b2fc0 con 0x7f90740b37c0 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f90794c4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9074158a70 0x7f90740b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f90794c4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9074158a70 0x7f90740b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48028/0 (socket says 172.21.15.77:48028) 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f90794c4640 1 -- 172.21.15.77:0/1110241718 learned_addr learned my addr 172.21.15.77:0/1110241718 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f90794c4640 1 -- 172.21.15.77:0/1110241718 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90740b37c0 msgr2=0x7f90740b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f9078cc3640 1 --2- 172.21.15.77:0/1110241718 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90740b37c0 0x7f90740b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f90794c4640 1 --2- 172.21.15.77:0/1110241718 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90740b37c0 0x7f90740b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f90794c4640 1 -- 172.21.15.77:0/1110241718 >> v1:172.21.15.121:6789/0 conn(0x7f907415b520 legacy=0x7f907415b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.665+0000 7f90794c4640 1 -- 172.21.15.77:0/1110241718 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90740b32a0 con 0x7f9074158a70 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.666+0000 7f90794c4640 1 --2- 172.21.15.77:0/1110241718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9074158a70 0x7f90740b40c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9070009870 tx=0x7f907002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec53df973bacab32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.666+0000 7f906bfff640 1 -- 172.21.15.77:0/1110241718 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9070037070 con 0x7f9074158a70 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.666+0000 7f906bfff640 1 -- 172.21.15.77:0/1110241718 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9070032440 con 0x7f9074158a70 2024-04-25T01:22:39.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.666+0000 7f906bfff640 1 -- 172.21.15.77:0/1110241718 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9070032810 con 0x7f9074158a70 2024-04-25T01:22:39.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.666+0000 7f907b728640 1 -- 172.21.15.77:0/1110241718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9074158a70 msgr2=0x7f90740b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.666+0000 7f907b728640 1 --2- 172.21.15.77:0/1110241718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9074158a70 0x7f90740b40c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9070009870 tx=0x7f907002a210 comp rx=0 tx=0).stop 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.667+0000 7f907b728640 1 -- 172.21.15.77:0/1110241718 shutdown_connections 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.667+0000 7f907b728640 1 --2- 172.21.15.77:0/1110241718 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90740b37c0 0x7f90740b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.667+0000 7f907b728640 1 --2- 172.21.15.77:0/1110241718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9074158a70 0x7f90740b40c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.667+0000 7f907b728640 1 -- 172.21.15.77:0/1110241718 >> 172.21.15.77:0/1110241718 conn(0x7f907406b720 msgr2=0x7f90740b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.667+0000 7f907b728640 1 -- 172.21.15.77:0/1110241718 shutdown_connections 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.667+0000 7f907b728640 1 -- 172.21.15.77:0/1110241718 wait complete. 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.668+0000 7f907b728640 1 Processor -- start 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.668+0000 7f907b728640 1 -- start start 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.668+0000 7f907b728640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90740b37c0 0x7f90741e4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.668+0000 7f907b728640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9074158a70 0x7f9074144660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.668+0000 7f907b728640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f907415b520 0x7f90740aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.668+0000 7f907b728640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f907406cb70 con 0x7f90740b37c0 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.668+0000 7f907b728640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f907406c9f0 con 0x7f907415b520 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.668+0000 7f907b728640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f907406ccf0 con 0x7f9074158a70 2024-04-25T01:22:39.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f90794c4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90740b37c0 0x7f90741e4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f90794c4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90740b37c0 0x7f90741e4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48036/0 (socket says 172.21.15.77:48036) 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f90794c4640 1 -- 172.21.15.77:0/3767802946 learned_addr learned my addr 172.21.15.77:0/3767802946 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f9078cc3640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9074158a70 0x7f9074144660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f9079cc5640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f907415b520 0x7f90740aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f90794c4640 1 -- 172.21.15.77:0/3767802946 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9074158a70 msgr2=0x7f9074144660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f90794c4640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9074158a70 0x7f9074144660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f90794c4640 1 -- 172.21.15.77:0/3767802946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f907415b520 msgr2=0x7f90740aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f90794c4640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f907415b520 0x7f90740aa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f90794c4640 1 -- 172.21.15.77:0/3767802946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90740aa620 con 0x7f90740b37c0 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f9079cc5640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f907415b520 0x7f90740aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f90794c4640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90740b37c0 0x7f90741e4930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9070009870 tx=0x7f90700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f9069ffb640 1 -- 172.21.15.77:0/3767802946 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9070041020 con 0x7f90740b37c0 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f9069ffb640 1 -- 172.21.15.77:0/3767802946 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9070030970 con 0x7f90740b37c0 2024-04-25T01:22:39.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f907b728640 1 -- 172.21.15.77:0/3767802946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90741e4f60 con 0x7f90740b37c0 2024-04-25T01:22:39.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f9069ffb640 1 -- 172.21.15.77:0/3767802946 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9070037040 con 0x7f90740b37c0 2024-04-25T01:22:39.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.669+0000 7f907b728640 1 -- 172.21.15.77:0/3767802946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90741e5520 con 0x7f90740b37c0 2024-04-25T01:22:39.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.670+0000 7f907b728640 1 -- 172.21.15.77:0/3767802946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9074068fb0 con 0x7f90740b37c0 2024-04-25T01:22:39.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.671+0000 7f9069ffb640 1 -- 172.21.15.77:0/3767802946 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9070044020 con 0x7f90740b37c0 2024-04-25T01:22:39.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.671+0000 7f9069ffb640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f905806be30 0x7f905806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.671+0000 7f9069ffb640 1 -- 172.21.15.77:0/3767802946 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f90700ab920 con 0x7f90740b37c0 2024-04-25T01:22:39.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.671+0000 7f9078cc3640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f905806be30 0x7f905806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.672+0000 7f9078cc3640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f905806be30 0x7f905806e2f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f907406d630 tx=0x7f9064008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.675+0000 7f9069ffb640 1 -- 172.21.15.77:0/3767802946 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f907004b070 con 0x7f90740b37c0 2024-04-25T01:22:39.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.852+0000 7f907b728640 1 -- 172.21.15.77:0/3767802946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f90740b3c70 con 0x7f90740b37c0 2024-04-25T01:22:39.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.852+0000 7f9069ffb640 1 -- 172.21.15.77:0/3767802946 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f9070079ec0 con 0x7f90740b37c0 2024-04-25T01:22:39.834 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 -- 172.21.15.77:0/3767802946 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f905806be30 msgr2=0x7f905806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f905806be30 0x7f905806e2f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f907406d630 tx=0x7f9064008040 comp rx=0 tx=0).stop 2024-04-25T01:22:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 -- 172.21.15.77:0/3767802946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90740b37c0 msgr2=0x7f90741e4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90740b37c0 0x7f90741e4930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9070009870 tx=0x7f90700046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 -- 172.21.15.77:0/3767802946 shutdown_connections 2024-04-25T01:22:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f905806be30 0x7f905806e2f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f907415b520 0x7f90740aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9074158a70 0x7f9074144660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 --2- 172.21.15.77:0/3767802946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90740b37c0 0x7f90741e4930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 -- 172.21.15.77:0/3767802946 >> 172.21.15.77:0/3767802946 conn(0x7f907406b720 msgr2=0x7f907415c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:39.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 -- 172.21.15.77:0/3767802946 shutdown_connections 2024-04-25T01:22:39.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.854+0000 7f907b728640 1 -- 172.21.15.77:0/3767802946 wait complete. 2024-04-25T01:22:39.847 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:39.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.964+0000 7f9129012640 1 Processor -- start 2024-04-25T01:22:39.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.964+0000 7f9129012640 1 -- start start 2024-04-25T01:22:39.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f9129012640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f91240d23d0 0x7f91240d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f9129012640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91240b9a90 0x7f91240d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f9129012640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91240cea40 con 0x7f91240d27b0 2024-04-25T01:22:39.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f9129012640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f91240cece0 con 0x7f91240d23d0 2024-04-25T01:22:39.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f9129012640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f91240cefc0 con 0x7f91240b9a90 2024-04-25T01:22:39.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f9122d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f91240d23d0 0x7f91240d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f912259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f91240d27b0 0x7f91240b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33000/0 (socket says 172.21.15.77:33000) 2024-04-25T01:22:39.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f912259c640 1 -- 172.21.15.77:0/733770426 learned_addr learned my addr 172.21.15.77:0/733770426 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:39.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f9122d9d640 1 -- 172.21.15.77:0/733770426 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91240b9a90 msgr2=0x7f91240d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:39.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f9122d9d640 1 --2- 172.21.15.77:0/733770426 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91240b9a90 0x7f91240d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f9122d9d640 1 -- 172.21.15.77:0/733770426 >> v1:172.21.15.77:6789/0 conn(0x7f91240d27b0 legacy=0x7f91240b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f9121d9b640 1 -- 172.21.15.77:0/733770426 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2752959113 0 0) 0x7f91240cea40 con 0x7f91240d27b0 2024-04-25T01:22:39.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.965+0000 7f9122d9d640 1 -- 172.21.15.77:0/733770426 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91240cf2a0 con 0x7f91240d23d0 2024-04-25T01:22:39.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.966+0000 7f9122d9d640 1 --2- 172.21.15.77:0/733770426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91240d23d0 0x7f91240d4920 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9114009d90 tx=0x7f911402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa5ab8d6caea9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.966+0000 7f9121d9b640 1 -- 172.21.15.77:0/733770426 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114007740 con 0x7f91240d23d0 2024-04-25T01:22:39.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.966+0000 7f9121d9b640 1 -- 172.21.15.77:0/733770426 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91140078e0 con 0x7f91240d23d0 2024-04-25T01:22:39.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.966+0000 7f9121d9b640 1 -- 172.21.15.77:0/733770426 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114007cb0 con 0x7f91240d23d0 2024-04-25T01:22:39.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.967+0000 7f9129012640 1 -- 172.21.15.77:0/733770426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91240d23d0 msgr2=0x7f91240d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.967+0000 7f9129012640 1 --2- 172.21.15.77:0/733770426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91240d23d0 0x7f91240d4920 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9114009d90 tx=0x7f911402a6f0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.967+0000 7f9129012640 1 -- 172.21.15.77:0/733770426 shutdown_connections 2024-04-25T01:22:39.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.967+0000 7f9129012640 1 --2- 172.21.15.77:0/733770426 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91240b9a90 0x7f91240d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.967+0000 7f9129012640 1 --2- 172.21.15.77:0/733770426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91240d23d0 0x7f91240d4920 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.967+0000 7f9129012640 1 -- 172.21.15.77:0/733770426 >> 172.21.15.77:0/733770426 conn(0x7f91240d3e30 msgr2=0x7f91240d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.967+0000 7f9129012640 1 -- 172.21.15.77:0/733770426 shutdown_connections 2024-04-25T01:22:39.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.967+0000 7f9129012640 1 -- 172.21.15.77:0/733770426 wait complete. 2024-04-25T01:22:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.968+0000 7f9129012640 1 Processor -- start 2024-04-25T01:22:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.968+0000 7f9129012640 1 -- start start 2024-04-25T01:22:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9129012640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91240b9a90 0x7f912414e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9129012640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91240d23d0 0x7f91240ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9129012640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91240d27b0 0x7f912415f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9129012640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f91240cfd80 con 0x7f91240b9a90 2024-04-25T01:22:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9129012640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f91240cf860 con 0x7f91240d23d0 2024-04-25T01:22:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9129012640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f91240b6c80 con 0x7f91240d27b0 2024-04-25T01:22:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9122d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91240b9a90 0x7f912414e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9122d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91240b9a90 0x7f912414e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48058/0 (socket says 172.21.15.77:48058) 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9122d9d640 1 -- 172.21.15.77:0/1286049721 learned_addr learned my addr 172.21.15.77:0/1286049721 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9122d9d640 1 -- 172.21.15.77:0/1286049721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91240d27b0 msgr2=0x7f912415f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9122d9d640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91240d27b0 0x7f912415f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9122d9d640 1 -- 172.21.15.77:0/1286049721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91240d23d0 msgr2=0x7f91240ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9122d9d640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91240d23d0 0x7f91240ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.969+0000 7f9122d9d640 1 -- 172.21.15.77:0/1286049721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9124158190 con 0x7f91240b9a90 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.970+0000 7f9122d9d640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91240b9a90 0x7f912414e040 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9114009d90 tx=0x7f9114004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.970+0000 7f910b7fe640 1 -- 172.21.15.77:0/1286049721 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114040070 con 0x7f91240b9a90 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.970+0000 7f910b7fe640 1 -- 172.21.15.77:0/1286049721 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9114031c40 con 0x7f91240b9a90 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.970+0000 7f9129012640 1 -- 172.21.15.77:0/1286049721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91240c9640 con 0x7f91240b9a90 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.970+0000 7f910b7fe640 1 -- 172.21.15.77:0/1286049721 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114003990 con 0x7f91240b9a90 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.970+0000 7f9129012640 1 -- 172.21.15.77:0/1286049721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91240c9c00 con 0x7f91240b9a90 2024-04-25T01:22:39.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.971+0000 7f9129012640 1 -- 172.21.15.77:0/1286049721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9124068fb0 con 0x7f91240b9a90 2024-04-25T01:22:39.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.971+0000 7f910b7fe640 1 -- 172.21.15.77:0/1286049721 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9114039040 con 0x7f91240b9a90 2024-04-25T01:22:39.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.972+0000 7f910b7fe640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f911006be30 0x7f911006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.972+0000 7f910b7fe640 1 -- 172.21.15.77:0/1286049721 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f91140ac980 con 0x7f91240b9a90 2024-04-25T01:22:39.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.972+0000 7f912259c640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f911006be30 0x7f911006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.972+0000 7f912259c640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f911006be30 0x7f911006e2f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9118000c00 tx=0x7f91180079c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:39.976+0000 7f910b7fe640 1 -- 172.21.15.77:0/1286049721 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f911407a8d0 con 0x7f91240b9a90 2024-04-25T01:22:40.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.148+0000 7f9129012640 1 -- 172.21.15.77:0/1286049721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f912408df50 con 0x7f91240b9a90 2024-04-25T01:22:40.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.149+0000 7f910b7fe640 1 -- 172.21.15.77:0/1286049721 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f911407ebd0 con 0x7f91240b9a90 2024-04-25T01:22:40.131 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:40.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.150+0000 7f9129012640 1 -- 172.21.15.77:0/1286049721 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f911006be30 msgr2=0x7f911006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.150+0000 7f9129012640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f911006be30 0x7f911006e2f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9118000c00 tx=0x7f91180079c0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.150+0000 7f9129012640 1 -- 172.21.15.77:0/1286049721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91240b9a90 msgr2=0x7f912414e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.150+0000 7f9129012640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91240b9a90 0x7f912414e040 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9114009d90 tx=0x7f9114004a40 comp rx=0 tx=0).stop 2024-04-25T01:22:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.150+0000 7f9129012640 1 -- 172.21.15.77:0/1286049721 shutdown_connections 2024-04-25T01:22:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.150+0000 7f9129012640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f911006be30 0x7f911006e2f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.150+0000 7f9129012640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91240d27b0 0x7f912415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.150+0000 7f9129012640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91240d23d0 0x7f91240ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.150+0000 7f9129012640 1 --2- 172.21.15.77:0/1286049721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91240b9a90 0x7f912414e040 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.150+0000 7f9129012640 1 -- 172.21.15.77:0/1286049721 >> 172.21.15.77:0/1286049721 conn(0x7f91240d3e30 msgr2=0x7f91240d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.151+0000 7f9129012640 1 -- 172.21.15.77:0/1286049721 shutdown_connections 2024-04-25T01:22:40.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.151+0000 7f9129012640 1 -- 172.21.15.77:0/1286049721 wait complete. 2024-04-25T01:22:40.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:40.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.261+0000 7ff454a27640 1 Processor -- start 2024-04-25T01:22:40.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.261+0000 7ff454a27640 1 -- start start 2024-04-25T01:22:40.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff454a27640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff45006caf0 0x7ff4500b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff454a27640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff45015bb80 con 0x7ff4500720c0 2024-04-25T01:22:40.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff454a27640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4500724a0 con 0x7ff45006da80 2024-04-25T01:22:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff454a27640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff4500b3750 con 0x7ff45006caf0 2024-04-25T01:22:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44e59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff4500720c0 0x7ff45006d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33016/0 (socket says 172.21.15.77:33016) 2024-04-25T01:22:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44e59c640 1 -- 172.21.15.77:0/3391859538 learned_addr learned my addr 172.21.15.77:0/3391859538 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2614674835 0 0) 0x7ff45015bb80 con 0x7ff4500720c0 2024-04-25T01:22:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff43c004630 con 0x7ff4500720c0 2024-04-25T01:22:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff447fff640 1 --2- 172.21.15.77:0/3391859538 >> v2:172.21.15.121:3300/0 conn(0x7ff45006caf0 0x7ff4500b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2317373048 0 0) 0x7ff43c004630 con 0x7ff4500720c0 2024-04-25T01:22:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 >> v2:172.21.15.121:3300/0 conn(0x7ff45006caf0 msgr2=0x7ff4500b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 --2- 172.21.15.77:0/3391859538 >> v2:172.21.15.121:3300/0 conn(0x7ff45006caf0 0x7ff4500b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 >> v1:172.21.15.121:6789/0 conn(0x7ff45006da80 legacy=0x7ff450070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4500b3a30 con 0x7ff4500720c0 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7ff440001670 con 0x7ff4500720c0 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 >> v1:172.21.15.77:6789/0 conn(0x7ff4500720c0 legacy=0x7ff45006d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff43c004ac0 0x7ff43c004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 0x7ff43c005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.262+0000 7ff44dd9b640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff43c0062d0 0x7ff43c00cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff4500b3750 con 0x7ff43c005910 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c003440 con 0x7ff43c0062d0 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff43c003640 con 0x7ff43c004ac0 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff447fff640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 0x7ff43c005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff44e59c640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff43c004ac0 0x7ff43c004ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff44ed9d640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff43c0062d0 0x7ff43c00cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff447fff640 1 -- 172.21.15.77:0/3391859538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff43c004ac0 msgr2=0x7ff43c004ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:40.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff447fff640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff43c004ac0 0x7ff43c004ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff447fff640 1 -- 172.21.15.77:0/3391859538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff43c0062d0 msgr2=0x7ff43c00cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff447fff640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff43c0062d0 0x7ff43c00cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff447fff640 1 -- 172.21.15.77:0/3391859538 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43c004630 con 0x7ff43c005910 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff44ed9d640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff43c0062d0 0x7ff43c00cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.263+0000 7ff44ed9d640 1 -- 172.21.15.77:0/3391859538 reap_dead start 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.264+0000 7ff447fff640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 0x7ff43c005d90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff438009a90 tx=0x7ff43c006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e20ec734e9cd53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.264+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438036070 con 0x7ff43c005910 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.264+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff438031400 con 0x7ff43c005910 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.264+0000 7ff44dd9b640 1 -- 172.21.15.77:0/3391859538 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438009690 con 0x7ff43c005910 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.264+0000 7ff454a27640 1 -- 172.21.15.77:0/3391859538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 msgr2=0x7ff43c005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.264+0000 7ff454a27640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 0x7ff43c005d90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff438009a90 tx=0x7ff43c006940 comp rx=0 tx=0).stop 2024-04-25T01:22:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.264+0000 7ff454a27640 1 -- 172.21.15.77:0/3391859538 shutdown_connections 2024-04-25T01:22:40.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.264+0000 7ff454a27640 1 --2- 172.21.15.77:0/3391859538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 0x7ff43c005d90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.264+0000 7ff454a27640 1 -- 172.21.15.77:0/3391859538 >> 172.21.15.77:0/3391859538 conn(0x7ff45006b720 msgr2=0x7ff4500b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:40.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.264+0000 7ff454a27640 1 -- 172.21.15.77:0/3391859538 shutdown_connections 2024-04-25T01:22:40.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.264+0000 7ff454a27640 1 -- 172.21.15.77:0/3391859538 wait complete. 2024-04-25T01:22:40.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.266+0000 7ff454a27640 1 Processor -- start 2024-04-25T01:22:40.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.266+0000 7ff454a27640 1 -- start start 2024-04-25T01:22:40.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.266+0000 7ff454a27640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 0x7ff45014c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.266+0000 7ff454a27640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff45015c750 0x7ff4500acab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.266+0000 7ff454a27640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff45015cbd0 0x7ff4500acff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.266+0000 7ff454a27640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff45015b6a0 con 0x7ff43c005910 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.266+0000 7ff454a27640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff4500b3ff0 con 0x7ff45015cbd0 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.266+0000 7ff454a27640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff45015b820 con 0x7ff45015c750 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.266+0000 7ff44e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 0x7ff45014c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.266+0000 7ff44e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 0x7ff45014c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48082/0 (socket says 172.21.15.77:48082) 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.266+0000 7ff44e59c640 1 -- 172.21.15.77:0/1587265315 learned_addr learned my addr 172.21.15.77:0/1587265315 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff447fff640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff45015c750 0x7ff4500acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff44e59c640 1 -- 172.21.15.77:0/1587265315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff45015c750 msgr2=0x7ff4500acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff44e59c640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff45015c750 0x7ff4500acab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff44e59c640 1 -- 172.21.15.77:0/1587265315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff45015cbd0 msgr2=0x7ff4500acff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff44e59c640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff45015cbd0 0x7ff4500acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff44e59c640 1 -- 172.21.15.77:0/1587265315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4500aa1e0 con 0x7ff43c005910 2024-04-25T01:22:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff447fff640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff45015c750 0x7ff4500acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff44e59c640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 0x7ff45014c530 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff4400072c0 tx=0x7ff440007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff446ffd640 1 -- 172.21.15.77:0/1587265315 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44000f860 con 0x7ff43c005910 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff446ffd640 1 -- 172.21.15.77:0/1587265315 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff440018070 con 0x7ff43c005910 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff446ffd640 1 -- 172.21.15.77:0/1587265315 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44000fc40 con 0x7ff43c005910 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff454a27640 1 -- 172.21.15.77:0/1587265315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff45015d170 con 0x7ff43c005910 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.267+0000 7ff454a27640 1 -- 172.21.15.77:0/1587265315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff45015d730 con 0x7ff43c005910 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.268+0000 7ff454a27640 1 -- 172.21.15.77:0/1587265315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff450068fb0 con 0x7ff43c005910 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.268+0000 7ff446ffd640 1 -- 172.21.15.77:0/1587265315 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff44001d460 con 0x7ff43c005910 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.269+0000 7ff446ffd640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff43006be60 0x7ff43006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.269+0000 7ff446ffd640 1 -- 172.21.15.77:0/1587265315 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7ff44008b340 con 0x7ff43c005910 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.269+0000 7ff447fff640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff43006be60 0x7ff43006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.269+0000 7ff447fff640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff43006be60 0x7ff43006e320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff438033000 tx=0x7ff438008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:40.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.273+0000 7ff446ffd640 1 -- 172.21.15.77:0/1587265315 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff4400598e0 con 0x7ff43c005910 2024-04-25T01:22:40.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.446+0000 7ff454a27640 1 -- 172.21.15.77:0/1587265315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff4500b3400 con 0x7ff43c005910 2024-04-25T01:22:40.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.446+0000 7ff446ffd640 1 -- 172.21.15.77:0/1587265315 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7ff4400591e0 con 0x7ff43c005910 2024-04-25T01:22:40.428 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:40.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 -- 172.21.15.77:0/1587265315 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff43006be60 msgr2=0x7ff43006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff43006be60 0x7ff43006e320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff438033000 tx=0x7ff438008030 comp rx=0 tx=0).stop 2024-04-25T01:22:40.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 -- 172.21.15.77:0/1587265315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 msgr2=0x7ff45014c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 0x7ff45014c530 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff4400072c0 tx=0x7ff440007780 comp rx=0 tx=0).stop 2024-04-25T01:22:40.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 -- 172.21.15.77:0/1587265315 shutdown_connections 2024-04-25T01:22:40.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff43006be60 0x7ff43006e320 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff45015cbd0 0x7ff4500acff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff45015c750 0x7ff4500acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 --2- 172.21.15.77:0/1587265315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff43c005910 0x7ff45014c530 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 -- 172.21.15.77:0/1587265315 >> 172.21.15.77:0/1587265315 conn(0x7ff45006b720 msgr2=0x7ff450157ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:40.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 -- 172.21.15.77:0/1587265315 shutdown_connections 2024-04-25T01:22:40.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.448+0000 7ff454a27640 1 -- 172.21.15.77:0/1587265315 wait complete. 2024-04-25T01:22:40.441 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:22:40.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.556+0000 7f648b645640 1 Processor -- start 2024-04-25T01:22:40.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.556+0000 7f648b645640 1 -- start start 2024-04-25T01:22:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.557+0000 7f648b645640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f648415e4c0 0x7f64840d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.557+0000 7f648b645640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64840d4a10 0x7f64840b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.557+0000 7f648b645640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64840d1e90 con 0x7f64840d2d40 2024-04-25T01:22:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.557+0000 7f648b645640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f64840b54a0 con 0x7f64840d4a10 2024-04-25T01:22:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.557+0000 7f648b645640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64840d2130 con 0x7f648415e4c0 2024-04-25T01:22:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.557+0000 7f6488be0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64840d4a10 0x7f64840b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.557+0000 7f6489be2640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f64840d2d40 0x7f64840d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33024/0 (socket says 172.21.15.77:33024) 2024-04-25T01:22:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.557+0000 7f6489be2640 1 -- 172.21.15.77:0/3200271945 learned_addr learned my addr 172.21.15.77:0/3200271945 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.558+0000 7f64893e1640 1 --2- 172.21.15.77:0/3200271945 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f648415e4c0 0x7f64840d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.558+0000 7f6488be0640 1 -- 172.21.15.77:0/3200271945 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f648415e4c0 msgr2=0x7f64840d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:40.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.558+0000 7f6488be0640 1 --2- 172.21.15.77:0/3200271945 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f648415e4c0 0x7f64840d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.558+0000 7f6488be0640 1 -- 172.21.15.77:0/3200271945 >> v1:172.21.15.77:6789/0 conn(0x7f64840d2d40 legacy=0x7f64840d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.558+0000 7f6488be0640 1 -- 172.21.15.77:0/3200271945 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64840b5780 con 0x7f64840d4a10 2024-04-25T01:22:40.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.558+0000 7f647bfff640 1 -- 172.21.15.77:0/3200271945 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3034577367 0 0) 0x7f64840d1e90 con 0x7f64840d2d40 2024-04-25T01:22:40.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.558+0000 7f6488be0640 1 --2- 172.21.15.77:0/3200271945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64840d4a10 0x7f64840b3ee0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f64740097a0 tx=0x7f647402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae0a45f990ebc1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:40.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.559+0000 7f647bfff640 1 -- 172.21.15.77:0/3200271945 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474037070 con 0x7f64840d4a10 2024-04-25T01:22:40.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.559+0000 7f647bfff640 1 -- 172.21.15.77:0/3200271945 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f647402ae50 con 0x7f64840d4a10 2024-04-25T01:22:40.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.559+0000 7f647bfff640 1 -- 172.21.15.77:0/3200271945 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474032660 con 0x7f64840d4a10 2024-04-25T01:22:40.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.559+0000 7f648b645640 1 -- 172.21.15.77:0/3200271945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64840d4a10 msgr2=0x7f64840b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.559+0000 7f648b645640 1 --2- 172.21.15.77:0/3200271945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64840d4a10 0x7f64840b3ee0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f64740097a0 tx=0x7f647402a100 comp rx=0 tx=0).stop 2024-04-25T01:22:40.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.559+0000 7f648b645640 1 -- 172.21.15.77:0/3200271945 shutdown_connections 2024-04-25T01:22:40.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.559+0000 7f648b645640 1 --2- 172.21.15.77:0/3200271945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64840d4a10 0x7f64840b3ee0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.559+0000 7f648b645640 1 --2- 172.21.15.77:0/3200271945 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f648415e4c0 0x7f64840d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.559+0000 7f648b645640 1 -- 172.21.15.77:0/3200271945 >> 172.21.15.77:0/3200271945 conn(0x7f64840d3ef0 msgr2=0x7f64840d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:40.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.559+0000 7f648b645640 1 -- 172.21.15.77:0/3200271945 shutdown_connections 2024-04-25T01:22:40.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.559+0000 7f648b645640 1 -- 172.21.15.77:0/3200271945 wait complete. 2024-04-25T01:22:40.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.560+0000 7f648b645640 1 Processor -- start 2024-04-25T01:22:40.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.560+0000 7f648b645640 1 -- start start 2024-04-25T01:22:40.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.561+0000 7f648b645640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64840d2d40 0x7f64841e8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.561+0000 7f648b645640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64840d4a10 0x7f64841e6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.561+0000 7f648b645640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f648415e4c0 0x7f64841e4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.561+0000 7f648b645640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f648415ea20 con 0x7f64840d2d40 2024-04-25T01:22:40.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.561+0000 7f648b645640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f648415e8a0 con 0x7f64840d4a10 2024-04-25T01:22:40.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.561+0000 7f648b645640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64840d1010 con 0x7f648415e4c0 2024-04-25T01:22:40.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.561+0000 7f64893e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64840d2d40 0x7f64841e8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.561+0000 7f64893e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64840d2d40 0x7f64841e8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48110/0 (socket says 172.21.15.77:48110) 2024-04-25T01:22:40.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.561+0000 7f64893e1640 1 -- 172.21.15.77:0/2673359332 learned_addr learned my addr 172.21.15.77:0/2673359332 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:40.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.561+0000 7f6489be2640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f648415e4c0 0x7f64841e4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.561+0000 7f6488be0640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64840d4a10 0x7f64841e6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f64893e1640 1 -- 172.21.15.77:0/2673359332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f648415e4c0 msgr2=0x7f64841e4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f64893e1640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f648415e4c0 0x7f64841e4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f64893e1640 1 -- 172.21.15.77:0/2673359332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64840d4a10 msgr2=0x7f64841e6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f64893e1640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64840d4a10 0x7f64841e6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f64893e1640 1 -- 172.21.15.77:0/2673359332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64841e71e0 con 0x7f64840d2d40 2024-04-25T01:22:40.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f6488be0640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64840d4a10 0x7f64841e6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:40.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f64893e1640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64840d2d40 0x7f64841e8720 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6480007e40 tx=0x7f648000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:40.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f6479ffb640 1 -- 172.21.15.77:0/2673359332 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480014620 con 0x7f64840d2d40 2024-04-25T01:22:40.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f6479ffb640 1 -- 172.21.15.77:0/2673359332 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6480019070 con 0x7f64840d2d40 2024-04-25T01:22:40.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f648b645640 1 -- 172.21.15.77:0/2673359332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64841e74c0 con 0x7f64840d2d40 2024-04-25T01:22:40.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f6479ffb640 1 -- 172.21.15.77:0/2673359332 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480014a00 con 0x7f64840d2d40 2024-04-25T01:22:40.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.562+0000 7f648b645640 1 -- 172.21.15.77:0/2673359332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64841e7a80 con 0x7f64840d2d40 2024-04-25T01:22:40.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.563+0000 7f648b645640 1 -- 172.21.15.77:0/2673359332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6484068fb0 con 0x7f64840d2d40 2024-04-25T01:22:40.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.563+0000 7f6479ffb640 1 -- 172.21.15.77:0/2673359332 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6480014ba0 con 0x7f64840d2d40 2024-04-25T01:22:40.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.564+0000 7f6479ffb640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f646806be30 0x7f646806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.564+0000 7f6479ffb640 1 -- 172.21.15.77:0/2673359332 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f648008ba60 con 0x7f64840d2d40 2024-04-25T01:22:40.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.564+0000 7f6488be0640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f646806be30 0x7f646806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.564+0000 7f6488be0640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f646806be30 0x7f646806e2f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6474034000 tx=0x7f647402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:40.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.568+0000 7f6479ffb640 1 -- 172.21.15.77:0/2673359332 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f648005a000 con 0x7f64840d2d40 2024-04-25T01:22:40.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.755+0000 7f648b645640 1 -- 172.21.15.77:0/2673359332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f64841e9a70 con 0x7f64840d2d40 2024-04-25T01:22:40.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.756+0000 7f6479ffb640 1 -- 172.21.15.77:0/2673359332 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f6480059900 con 0x7f64840d2d40 2024-04-25T01:22:40.737 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-04-25T01:22:40.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 -- 172.21.15.77:0/2673359332 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f646806be30 msgr2=0x7f646806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f646806be30 0x7f646806e2f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6474034000 tx=0x7f647402e000 comp rx=0 tx=0).stop 2024-04-25T01:22:40.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 -- 172.21.15.77:0/2673359332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64840d2d40 msgr2=0x7f64841e8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64840d2d40 0x7f64841e8720 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6480007e40 tx=0x7f648000e4b0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 -- 172.21.15.77:0/2673359332 shutdown_connections 2024-04-25T01:22:40.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f646806be30 0x7f646806e2f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f648415e4c0 0x7f64841e4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64840d4a10 0x7f64841e6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 --2- 172.21.15.77:0/2673359332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64840d2d40 0x7f64841e8720 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 -- 172.21.15.77:0/2673359332 >> 172.21.15.77:0/2673359332 conn(0x7f64840d3ef0 msgr2=0x7f648406c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:40.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 -- 172.21.15.77:0/2673359332 shutdown_connections 2024-04-25T01:22:40.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:40.758+0000 7f648b645640 1 -- 172.21.15.77:0/2673359332 wait complete. 2024-04-25T01:22:40.750 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-04-25T01:22:40.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:40.751 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-04-25T01:22:40.775 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:22:40.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:40.776 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-25T01:22:40.831 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:22:40.831 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:40.851 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:22:40.851 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:40.909 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:22:40.909 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:22:40.967 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:41.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.083+0000 7f1f195b6640 1 Processor -- start 2024-04-25T01:22:41.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.083+0000 7f1f195b6640 1 -- start start 2024-04-25T01:22:41.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.084+0000 7f1f195b6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f1415c090 0x7f1f140b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.084+0000 7f1f195b6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f140cf9f0 con 0x7f1f140d27a0 2024-04-25T01:22:41.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.084+0000 7f1f195b6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f140cf0c0 con 0x7f1f140b0d80 2024-04-25T01:22:41.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.084+0000 7f1f195b6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f140d2c30 con 0x7f1f1415c090 2024-04-25T01:22:41.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.084+0000 7f1f137fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1f140d27a0 0x7f1f140cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33036/0 (socket says 172.21.15.77:33036) 2024-04-25T01:22:41.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.084+0000 7f1f137fe640 1 -- 172.21.15.77:0/3440405240 learned_addr learned my addr 172.21.15.77:0/3440405240 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.084+0000 7f1f12ffd640 1 --2- 172.21.15.77:0/3440405240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f1415c090 0x7f1f140b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.085+0000 7f1f127fc640 1 -- 172.21.15.77:0/3440405240 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1561007003 0 0) 0x7f1f140cf9f0 con 0x7f1f140d27a0 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.085+0000 7f1f127fc640 1 -- 172.21.15.77:0/3440405240 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f00003520 con 0x7f1f140d27a0 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.085+0000 7f1f127fc640 1 -- 172.21.15.77:0/3440405240 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3650825695 0 0) 0x7f1f140cf0c0 con 0x7f1f140b0d80 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.085+0000 7f1f127fc640 1 -- 172.21.15.77:0/3440405240 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f140cf9f0 con 0x7f1f140b0d80 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.085+0000 7f1f12ffd640 1 -- 172.21.15.77:0/3440405240 >> v1:172.21.15.121:6789/0 conn(0x7f1f140b0d80 legacy=0x7f1f140cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.085+0000 7f1f12ffd640 1 -- 172.21.15.77:0/3440405240 >> v1:172.21.15.77:6789/0 conn(0x7f1f140d27a0 legacy=0x7f1f140cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.085+0000 7f1f12ffd640 1 -- 172.21.15.77:0/3440405240 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f140cf360 con 0x7f1f1415c090 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.085+0000 7f1f12ffd640 1 --2- 172.21.15.77:0/3440405240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f1415c090 0x7f1f140b1180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1f040097d0 tx=0x7f1f0402a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c434da49c2fe0f7e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.086+0000 7f1f127fc640 1 -- 172.21.15.77:0/3440405240 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04037070 con 0x7f1f1415c090 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.086+0000 7f1f127fc640 1 -- 172.21.15.77:0/3440405240 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f04032440 con 0x7f1f1415c090 2024-04-25T01:22:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.086+0000 7f1f127fc640 1 -- 172.21.15.77:0/3440405240 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f040327f0 con 0x7f1f1415c090 2024-04-25T01:22:41.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.086+0000 7f1f195b6640 1 -- 172.21.15.77:0/3440405240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f1415c090 msgr2=0x7f1f140b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.086+0000 7f1f195b6640 1 --2- 172.21.15.77:0/3440405240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f1415c090 0x7f1f140b1180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1f040097d0 tx=0x7f1f0402a290 comp rx=0 tx=0).stop 2024-04-25T01:22:41.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.086+0000 7f1f195b6640 1 -- 172.21.15.77:0/3440405240 shutdown_connections 2024-04-25T01:22:41.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.086+0000 7f1f195b6640 1 --2- 172.21.15.77:0/3440405240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f1415c090 0x7f1f140b1180 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.086+0000 7f1f195b6640 1 -- 172.21.15.77:0/3440405240 >> 172.21.15.77:0/3440405240 conn(0x7f1f140d1c80 msgr2=0x7f1f140d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:41.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.086+0000 7f1f195b6640 1 -- 172.21.15.77:0/3440405240 shutdown_connections 2024-04-25T01:22:41.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.086+0000 7f1f195b6640 1 -- 172.21.15.77:0/3440405240 wait complete. 2024-04-25T01:22:41.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.087+0000 7f1f195b6640 1 Processor -- start 2024-04-25T01:22:41.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f195b6640 1 -- start start 2024-04-25T01:22:41.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f195b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f140b0d80 0x7f1f1406dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f195b6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f1415c090 0x7f1f140c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f195b6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f141f3c90 0x7f1f140cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f195b6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1f140d0620 con 0x7f1f140b0d80 2024-04-25T01:22:41.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f195b6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1415c470 con 0x7f1f141f3c90 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f195b6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f140d07a0 con 0x7f1f1415c090 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f12ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f140b0d80 0x7f1f1406dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f12ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f140b0d80 0x7f1f1406dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48112/0 (socket says 172.21.15.77:48112) 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f12ffd640 1 -- 172.21.15.77:0/1255987274 learned_addr learned my addr 172.21.15.77:0/1255987274 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f0a7fc640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f1415c090 0x7f1f140c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.088+0000 7f1f137fe640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f141f3c90 0x7f1f140cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.089+0000 7f1f12ffd640 1 -- 172.21.15.77:0/1255987274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f1415c090 msgr2=0x7f1f140c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.089+0000 7f1f12ffd640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f1415c090 0x7f1f140c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.089+0000 7f1f12ffd640 1 -- 172.21.15.77:0/1255987274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f141f3c90 msgr2=0x7f1f140cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.089+0000 7f1f12ffd640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f141f3c90 0x7f1f140cadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.089+0000 7f1f12ffd640 1 -- 172.21.15.77:0/1255987274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f1406e230 con 0x7f1f140b0d80 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.089+0000 7f1f12ffd640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f140b0d80 0x7f1f1406dcf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1f0402a2d0 tx=0x7f1f040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.089+0000 7f1f0bfff640 1 -- 172.21.15.77:0/1255987274 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04041030 con 0x7f1f140b0d80 2024-04-25T01:22:41.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.089+0000 7f1f0bfff640 1 -- 172.21.15.77:0/1255987274 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f0403d420 con 0x7f1f140b0d80 2024-04-25T01:22:41.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.089+0000 7f1f0bfff640 1 -- 172.21.15.77:0/1255987274 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04037040 con 0x7f1f140b0d80 2024-04-25T01:22:41.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.089+0000 7f1f195b6640 1 -- 172.21.15.77:0/1255987274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f140caae0 con 0x7f1f140b0d80 2024-04-25T01:22:41.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.089+0000 7f1f195b6640 1 -- 172.21.15.77:0/1255987274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f141f4470 con 0x7f1f140b0d80 2024-04-25T01:22:41.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.090+0000 7f1f195b6640 1 -- 172.21.15.77:0/1255987274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f14068180 con 0x7f1f140b0d80 2024-04-25T01:22:41.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.091+0000 7f1f0bfff640 1 -- 172.21.15.77:0/1255987274 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1f04044070 con 0x7f1f140b0d80 2024-04-25T01:22:41.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.091+0000 7f1f0bfff640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ef406be30 0x7f1ef406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.091+0000 7f1f0bfff640 1 -- 172.21.15.77:0/1255987274 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f1f040ab8b0 con 0x7f1f140b0d80 2024-04-25T01:22:41.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.091+0000 7f1f0a7fc640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ef406be30 0x7f1ef406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.091+0000 7f1f0a7fc640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ef406be30 0x7f1ef406e2f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1f1415fb50 tx=0x7f1efc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.095+0000 7f1f0bfff640 1 -- 172.21.15.77:0/1255987274 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1f04079e60 con 0x7f1f140b0d80 2024-04-25T01:22:41.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.270+0000 7f1f195b6640 1 -- 172.21.15.77:0/1255987274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f140c9290 con 0x7f1f140b0d80 2024-04-25T01:22:41.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.271+0000 7f1f0bfff640 1 -- 172.21.15.77:0/1255987274 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18922 (secure 0 0 0) 0x7f1f04079760 con 0x7f1f140b0d80 2024-04-25T01:22:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:41.253 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:26.287128+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:41.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 -- 172.21.15.77:0/1255987274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ef406be30 msgr2=0x7f1ef406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ef406be30 0x7f1ef406e2f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1f1415fb50 tx=0x7f1efc009040 comp rx=0 tx=0).stop 2024-04-25T01:22:41.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 -- 172.21.15.77:0/1255987274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f140b0d80 msgr2=0x7f1f1406dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f140b0d80 0x7f1f1406dcf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1f0402a2d0 tx=0x7f1f040046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 -- 172.21.15.77:0/1255987274 shutdown_connections 2024-04-25T01:22:41.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ef406be30 0x7f1ef406e2f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f141f3c90 0x7f1f140cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f1415c090 0x7f1f140c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 --2- 172.21.15.77:0/1255987274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f140b0d80 0x7f1f1406dcf0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 -- 172.21.15.77:0/1255987274 >> 172.21.15.77:0/1255987274 conn(0x7f1f140d1c80 msgr2=0x7f1f140b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:41.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 -- 172.21.15.77:0/1255987274 shutdown_connections 2024-04-25T01:22:41.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.273+0000 7f1f195b6640 1 -- 172.21.15.77:0/1255987274 wait complete. 2024-04-25T01:22:41.266 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:41.267 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:41.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.386+0000 7fecae5dc640 1 Processor -- start 2024-04-25T01:22:41.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.386+0000 7fecae5dc640 1 -- start start 2024-04-25T01:22:41.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.387+0000 7fecae5dc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca80d23d0 0x7feca80d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.387+0000 7fecae5dc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7feca80d27b0 0x7feca80d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.387+0000 7fecae5dc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feca80cea40 con 0x7feca80b9380 2024-04-25T01:22:41.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.387+0000 7fecae5dc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7feca80cece0 con 0x7feca80d27b0 2024-04-25T01:22:41.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.387+0000 7fecae5dc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7feca80cefc0 con 0x7feca80d23d0 2024-04-25T01:22:41.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.387+0000 7fecacb79640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7feca80b9380 0x7feca80b97f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33052/0 (socket says 172.21.15.77:33052) 2024-04-25T01:22:41.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.387+0000 7fecacb79640 1 -- 172.21.15.77:0/3666746453 learned_addr learned my addr 172.21.15.77:0/3666746453 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:41.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 117193103 0 0) 0x7feca80cea40 con 0x7feca80b9380 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec90003520 con 0x7feca80b9380 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca7fff640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca80d23d0 0x7feca80d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca77fe640 1 --2- 172.21.15.77:0/3666746453 >> v2:172.21.15.121:3300/0 conn(0x7feca80d27b0 0x7feca80d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4197039253 0 0) 0x7fec90003520 con 0x7feca80b9380 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca80d23d0 msgr2=0x7feca80d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca80d23d0 0x7feca80d4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 >> v2:172.21.15.121:3300/0 conn(0x7feca80d27b0 msgr2=0x7feca80d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 --2- 172.21.15.77:0/3666746453 >> v2:172.21.15.121:3300/0 conn(0x7feca80d27b0 0x7feca80d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca80cf2a0 con 0x7feca80b9380 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7feca00019f0 con 0x7feca80b9380 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 >> v1:172.21.15.77:6789/0 conn(0x7feca80b9380 legacy=0x7feca80b97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900039b0 0x7fec90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec90004800 0x7fec90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca80d23d0 0x7fec900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feca80cece0 con 0x7fec900039b0 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feca80cefc0 con 0x7fec90004800 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec9000b9c0 con 0x7feca80d23d0 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca7fff640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900039b0 0x7fec90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7fecacb79640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca80d23d0 0x7fec900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca77fe640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec90004800 0x7fec90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca7fff640 1 -- 172.21.15.77:0/3666746453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca80d23d0 msgr2=0x7fec900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca7fff640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca80d23d0 0x7fec900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca7fff640 1 -- 172.21.15.77:0/3666746453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec90004800 msgr2=0x7fec90004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca7fff640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec90004800 0x7fec90004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.388+0000 7feca7fff640 1 -- 172.21.15.77:0/3666746453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec90003520 con 0x7fec900039b0 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.389+0000 7feca7fff640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900039b0 0x7fec90003dd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fec98009ac0 tx=0x7fec90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5400a6629e865 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.389+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec98036070 con 0x7fec900039b0 2024-04-25T01:22:41.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.389+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec98031400 con 0x7fec900039b0 2024-04-25T01:22:41.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.389+0000 7feca6ffd640 1 -- 172.21.15.77:0/3666746453 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec98009690 con 0x7fec900039b0 2024-04-25T01:22:41.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.389+0000 7fecae5dc640 1 -- 172.21.15.77:0/3666746453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900039b0 msgr2=0x7fec90003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.389+0000 7fecae5dc640 1 --2- 172.21.15.77:0/3666746453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900039b0 0x7fec90003dd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fec98009ac0 tx=0x7fec90005830 comp rx=0 tx=0).stop 2024-04-25T01:22:41.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.390+0000 7fecacb79640 1 -- 172.21.15.77:0/3666746453 reap_dead start 2024-04-25T01:22:41.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.390+0000 7fecae5dc640 1 -- 172.21.15.77:0/3666746453 shutdown_connections 2024-04-25T01:22:41.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.390+0000 7fecae5dc640 1 -- 172.21.15.77:0/3666746453 >> 172.21.15.77:0/3666746453 conn(0x7feca80d3e30 msgr2=0x7feca80d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:41.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.390+0000 7fecae5dc640 1 -- 172.21.15.77:0/3666746453 shutdown_connections 2024-04-25T01:22:41.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.390+0000 7fecae5dc640 1 -- 172.21.15.77:0/3666746453 wait complete. 2024-04-25T01:22:41.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.391+0000 7fecae5dc640 1 Processor -- start 2024-04-25T01:22:41.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.391+0000 7fecae5dc640 1 -- start start 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecae5dc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec900039b0 0x7feca81f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecae5dc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca81f44e0 0x7feca81f67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecae5dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca80b6a50 0x7feca80b6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecae5dc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feca80cfd80 con 0x7feca80b6a50 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecae5dc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feca80cf860 con 0x7fec900039b0 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecae5dc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7feca80d4920 con 0x7feca81f44e0 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecacb79640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca80b6a50 0x7feca80b6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecacb79640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca80b6a50 0x7feca80b6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48132/0 (socket says 172.21.15.77:48132) 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecacb79640 1 -- 172.21.15.77:0/3779022105 learned_addr learned my addr 172.21.15.77:0/3779022105 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7feca77fe640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca81f44e0 0x7feca81f67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7feca7fff640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec900039b0 0x7feca81f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecacb79640 1 -- 172.21.15.77:0/3779022105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca81f44e0 msgr2=0x7feca81f67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecacb79640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca81f44e0 0x7feca81f67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecacb79640 1 -- 172.21.15.77:0/3779022105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec900039b0 msgr2=0x7feca81f4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecacb79640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec900039b0 0x7feca81f4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecacb79640 1 -- 172.21.15.77:0/3779022105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca809ea60 con 0x7feca80b6a50 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecacb79640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca80b6a50 0x7feca80b6ed0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feca0002070 tx=0x7feca000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7feca4ff9640 1 -- 172.21.15.77:0/3779022105 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca000ee20 con 0x7feca80b6a50 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7feca4ff9640 1 -- 172.21.15.77:0/3779022105 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feca0018070 con 0x7feca80b6a50 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecae5dc640 1 -- 172.21.15.77:0/3779022105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca809ed00 con 0x7feca80b6a50 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7feca4ff9640 1 -- 172.21.15.77:0/3779022105 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0013660 con 0x7feca80b6a50 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.392+0000 7fecae5dc640 1 -- 172.21.15.77:0/3779022105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca809f2c0 con 0x7feca80b6a50 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.393+0000 7fecae5dc640 1 -- 172.21.15.77:0/3779022105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feca8068fb0 con 0x7feca80b6a50 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.394+0000 7feca4ff9640 1 -- 172.21.15.77:0/3779022105 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7feca0013800 con 0x7feca80b6a50 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.394+0000 7feca4ff9640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec9406be60 0x7fec9406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.394+0000 7feca4ff9640 1 -- 172.21.15.77:0/3779022105 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7feca008b4b0 con 0x7feca80b6a50 2024-04-25T01:22:41.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.394+0000 7feca7fff640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec9406be60 0x7fec9406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.395+0000 7feca7fff640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec9406be60 0x7fec9406e320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fec98033000 tx=0x7fec98008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.398+0000 7feca4ff9640 1 -- 172.21.15.77:0/3779022105 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7feca0059400 con 0x7feca80b6a50 2024-04-25T01:22:41.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.572+0000 7fecae5dc640 1 -- 172.21.15.77:0/3779022105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7feca808df50 con 0x7feca80b6a50 2024-04-25T01:22:41.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.572+0000 7feca4ff9640 1 -- 172.21.15.77:0/3779022105 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7feca005d700 con 0x7feca80b6a50 2024-04-25T01:22:41.554 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:41.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.574+0000 7fecae5dc640 1 -- 172.21.15.77:0/3779022105 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec9406be60 msgr2=0x7fec9406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.574+0000 7fecae5dc640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec9406be60 0x7fec9406e320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fec98033000 tx=0x7fec98008030 comp rx=0 tx=0).stop 2024-04-25T01:22:41.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.575+0000 7fecae5dc640 1 -- 172.21.15.77:0/3779022105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca80b6a50 msgr2=0x7feca80b6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.575+0000 7fecae5dc640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca80b6a50 0x7feca80b6ed0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feca0002070 tx=0x7feca000e530 comp rx=0 tx=0).stop 2024-04-25T01:22:41.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.575+0000 7fecae5dc640 1 -- 172.21.15.77:0/3779022105 shutdown_connections 2024-04-25T01:22:41.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.575+0000 7fecae5dc640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fec9406be60 0x7fec9406e320 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.575+0000 7fecae5dc640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca80b6a50 0x7feca80b6ed0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.575+0000 7fecae5dc640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feca81f44e0 0x7feca81f67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.575+0000 7fecae5dc640 1 --2- 172.21.15.77:0/3779022105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec900039b0 0x7feca81f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.575+0000 7fecae5dc640 1 -- 172.21.15.77:0/3779022105 >> 172.21.15.77:0/3779022105 conn(0x7feca80d3e30 msgr2=0x7feca806d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:41.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.575+0000 7fecae5dc640 1 -- 172.21.15.77:0/3779022105 shutdown_connections 2024-04-25T01:22:41.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.575+0000 7fecae5dc640 1 -- 172.21.15.77:0/3779022105 wait complete. 2024-04-25T01:22:41.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.686+0000 7f01cb54f640 1 Processor -- start 2024-04-25T01:22:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.686+0000 7f01cb54f640 1 -- start start 2024-04-25T01:22:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01cb54f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01c415fc40 0x7f01c41600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01cb54f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01c40cedc0 0x7f01c40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01cb54f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c40cf770 con 0x7f01c40722e0 2024-04-25T01:22:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01cb54f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f01c4160aa0 con 0x7f01c415fc40 2024-04-25T01:22:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01cb54f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f01c406da40 con 0x7f01c40cedc0 2024-04-25T01:22:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01c8aea640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01c415fc40 0x7f01c41600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01c92eb640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f01c40722e0 0x7f01c406ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33064/0 (socket says 172.21.15.77:33064) 2024-04-25T01:22:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01c92eb640 1 -- 172.21.15.77:0/1131002947 learned_addr learned my addr 172.21.15.77:0/1131002947 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01c9aec640 1 --2- 172.21.15.77:0/1131002947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01c40cedc0 0x7f01c40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01c8aea640 1 -- 172.21.15.77:0/1131002947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01c40cedc0 msgr2=0x7f01c40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01c8aea640 1 --2- 172.21.15.77:0/1131002947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01c40cedc0 0x7f01c40cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01c8aea640 1 -- 172.21.15.77:0/1131002947 >> v1:172.21.15.77:6789/0 conn(0x7f01c40722e0 legacy=0x7f01c406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01c3fff640 1 -- 172.21.15.77:0/1131002947 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2984273101 0 0) 0x7f01c40cf770 con 0x7f01c40722e0 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.687+0000 7f01c8aea640 1 -- 172.21.15.77:0/1131002947 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c406d110 con 0x7f01c415fc40 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.688+0000 7f01c8aea640 1 --2- 172.21.15.77:0/1131002947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c415fc40 0x7f01c41600a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f01b80097a0 tx=0x7f01b802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4d46dad79a5721 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.688+0000 7f01c3fff640 1 -- 172.21.15.77:0/1131002947 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b8037070 con 0x7f01c415fc40 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.688+0000 7f01c3fff640 1 -- 172.21.15.77:0/1131002947 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01b802ae50 con 0x7f01c415fc40 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.688+0000 7f01c3fff640 1 -- 172.21.15.77:0/1131002947 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b8032660 con 0x7f01c415fc40 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.689+0000 7f01cb54f640 1 -- 172.21.15.77:0/1131002947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c415fc40 msgr2=0x7f01c41600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.689+0000 7f01cb54f640 1 --2- 172.21.15.77:0/1131002947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c415fc40 0x7f01c41600a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f01b80097a0 tx=0x7f01b802a100 comp rx=0 tx=0).stop 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.689+0000 7f01cb54f640 1 -- 172.21.15.77:0/1131002947 shutdown_connections 2024-04-25T01:22:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.689+0000 7f01cb54f640 1 --2- 172.21.15.77:0/1131002947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01c40cedc0 0x7f01c40cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.689+0000 7f01cb54f640 1 --2- 172.21.15.77:0/1131002947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c415fc40 0x7f01c41600a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.689+0000 7f01cb54f640 1 -- 172.21.15.77:0/1131002947 >> 172.21.15.77:0/1131002947 conn(0x7f01c406b720 msgr2=0x7f01c40b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.689+0000 7f01cb54f640 1 -- 172.21.15.77:0/1131002947 shutdown_connections 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.689+0000 7f01cb54f640 1 -- 172.21.15.77:0/1131002947 wait complete. 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.690+0000 7f01cb54f640 1 Processor -- start 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.690+0000 7f01cb54f640 1 -- start start 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01cb54f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01c40722e0 0x7f01c4144820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01cb54f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c40cedc0 0x7f01c41417a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01cb54f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01c415fc40 0x7f01c40b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01cb54f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f01c406cec0 con 0x7f01c40cedc0 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01cb54f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f01c406d6d0 con 0x7f01c40722e0 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01cb54f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f01c4070b50 con 0x7f01c415fc40 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c8aea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c40cedc0 0x7f01c41417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c8aea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c40cedc0 0x7f01c41417a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48146/0 (socket says 172.21.15.77:48146) 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c8aea640 1 -- 172.21.15.77:0/2992390655 learned_addr learned my addr 172.21.15.77:0/2992390655 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c8aea640 1 -- 172.21.15.77:0/2992390655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01c415fc40 msgr2=0x7f01c40b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c8aea640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01c415fc40 0x7f01c40b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c8aea640 1 -- 172.21.15.77:0/2992390655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01c40722e0 msgr2=0x7f01c4144820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c92eb640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01c40722e0 0x7f01c4144820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c8aea640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01c40722e0 0x7f01c4144820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c8aea640 1 -- 172.21.15.77:0/2992390655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c4143500 con 0x7f01c40cedc0 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c8aea640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c40cedc0 0x7f01c41417a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f01b80098c0 tx=0x7f01b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c1ffb640 1 -- 172.21.15.77:0/2992390655 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b8041030 con 0x7f01c40cedc0 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c1ffb640 1 -- 172.21.15.77:0/2992390655 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01b802ac50 con 0x7f01c40cedc0 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01cb54f640 1 -- 172.21.15.77:0/2992390655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c41437e0 con 0x7f01c40cedc0 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.691+0000 7f01c1ffb640 1 -- 172.21.15.77:0/2992390655 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b8037040 con 0x7f01c40cedc0 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.692+0000 7f01cb54f640 1 -- 172.21.15.77:0/2992390655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c4143da0 con 0x7f01c40cedc0 2024-04-25T01:22:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.692+0000 7f01cb54f640 1 -- 172.21.15.77:0/2992390655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c4068fb0 con 0x7f01c40cedc0 2024-04-25T01:22:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.693+0000 7f01c1ffb640 1 -- 172.21.15.77:0/2992390655 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f01b8044070 con 0x7f01c40cedc0 2024-04-25T01:22:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.693+0000 7f01c1ffb640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01b006be60 0x7f01b006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.693+0000 7f01c1ffb640 1 -- 172.21.15.77:0/2992390655 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f01b80ab980 con 0x7f01c40cedc0 2024-04-25T01:22:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.693+0000 7f01c92eb640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01b006be60 0x7f01b006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.694+0000 7f01c92eb640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01b006be60 0x7f01b006e320 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f01b4007fd0 tx=0x7f01b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.697+0000 7f01c1ffb640 1 -- 172.21.15.77:0/2992390655 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f01b8044e40 con 0x7f01c40cedc0 2024-04-25T01:22:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.868+0000 7f01cb54f640 1 -- 172.21.15.77:0/2992390655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f01c4072c90 con 0x7f01c40cedc0 2024-04-25T01:22:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.868+0000 7f01c1ffb640 1 -- 172.21.15.77:0/2992390655 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f01b807a040 con 0x7f01c40cedc0 2024-04-25T01:22:41.850 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.870+0000 7f01cb54f640 1 -- 172.21.15.77:0/2992390655 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01b006be60 msgr2=0x7f01b006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.870+0000 7f01cb54f640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01b006be60 0x7f01b006e320 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f01b4007fd0 tx=0x7f01b4009040 comp rx=0 tx=0).stop 2024-04-25T01:22:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.870+0000 7f01cb54f640 1 -- 172.21.15.77:0/2992390655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c40cedc0 msgr2=0x7f01c41417a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.870+0000 7f01cb54f640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c40cedc0 0x7f01c41417a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f01b80098c0 tx=0x7f01b80046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.870+0000 7f01cb54f640 1 -- 172.21.15.77:0/2992390655 shutdown_connections 2024-04-25T01:22:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.870+0000 7f01cb54f640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f01b006be60 0x7f01b006e320 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.870+0000 7f01cb54f640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01c40cedc0 0x7f01c41417a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.870+0000 7f01cb54f640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01c415fc40 0x7f01c40b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.870+0000 7f01cb54f640 1 --2- 172.21.15.77:0/2992390655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01c40722e0 0x7f01c4144820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.870+0000 7f01cb54f640 1 -- 172.21.15.77:0/2992390655 >> 172.21.15.77:0/2992390655 conn(0x7f01c406b720 msgr2=0x7f01c40ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.871+0000 7f01cb54f640 1 -- 172.21.15.77:0/2992390655 shutdown_connections 2024-04-25T01:22:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.871+0000 7f01cb54f640 1 -- 172.21.15.77:0/2992390655 wait complete. 2024-04-25T01:22:41.863 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:41.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.981+0000 7fbb94e37640 1 Processor -- start 2024-04-25T01:22:41.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.981+0000 7fbb94e37640 1 -- start start 2024-04-25T01:22:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb94e37640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb90158a70 0x7fbb900b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb94e37640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbb9015bc30 0x7fbb900b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb94e37640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb9015c210 con 0x7fbb900b37c0 2024-04-25T01:22:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb94e37640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fbb900b2ce0 con 0x7fbb90158a70 2024-04-25T01:22:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb94e37640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbb900b2fc0 con 0x7fbb9015bc30 2024-04-25T01:22:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb8e59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb90158a70 0x7fbb900b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb8e59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb90158a70 0x7fbb900b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48152/0 (socket says 172.21.15.77:48152) 2024-04-25T01:22:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb8e59c640 1 -- 172.21.15.77:0/3263824987 learned_addr learned my addr 172.21.15.77:0/3263824987 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb8e59c640 1 -- 172.21.15.77:0/3263824987 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbb9015bc30 msgr2=0x7fbb900b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb8e59c640 1 --2- 172.21.15.77:0/3263824987 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbb9015bc30 0x7fbb900b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb8e59c640 1 -- 172.21.15.77:0/3263824987 >> v1:172.21.15.121:6789/0 conn(0x7fbb900b37c0 legacy=0x7fbb9015b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:41.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.982+0000 7fbb8e59c640 1 -- 172.21.15.77:0/3263824987 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb900b32a0 con 0x7fbb90158a70 2024-04-25T01:22:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.983+0000 7fbb8e59c640 1 --2- 172.21.15.77:0/3263824987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb90158a70 0x7fbb900b40c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbb80009e80 tx=0x7fbb8002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6c88bb015a0585 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.983+0000 7fbb8d59a640 1 -- 172.21.15.77:0/3263824987 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb80036070 con 0x7fbb90158a70 2024-04-25T01:22:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.983+0000 7fbb8d59a640 1 -- 172.21.15.77:0/3263824987 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb80031600 con 0x7fbb90158a70 2024-04-25T01:22:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.983+0000 7fbb8d59a640 1 -- 172.21.15.77:0/3263824987 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb800319d0 con 0x7fbb90158a70 2024-04-25T01:22:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.984+0000 7fbb94e37640 1 -- 172.21.15.77:0/3263824987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb90158a70 msgr2=0x7fbb900b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.984+0000 7fbb94e37640 1 --2- 172.21.15.77:0/3263824987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb90158a70 0x7fbb900b40c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbb80009e80 tx=0x7fbb8002a660 comp rx=0 tx=0).stop 2024-04-25T01:22:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.984+0000 7fbb94e37640 1 -- 172.21.15.77:0/3263824987 shutdown_connections 2024-04-25T01:22:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.984+0000 7fbb94e37640 1 --2- 172.21.15.77:0/3263824987 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbb9015bc30 0x7fbb900b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.984+0000 7fbb94e37640 1 --2- 172.21.15.77:0/3263824987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb90158a70 0x7fbb900b40c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.984+0000 7fbb94e37640 1 -- 172.21.15.77:0/3263824987 >> 172.21.15.77:0/3263824987 conn(0x7fbb9006b720 msgr2=0x7fbb900b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.984+0000 7fbb94e37640 1 -- 172.21.15.77:0/3263824987 shutdown_connections 2024-04-25T01:22:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.984+0000 7fbb94e37640 1 -- 172.21.15.77:0/3263824987 wait complete. 2024-04-25T01:22:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.985+0000 7fbb94e37640 1 Processor -- start 2024-04-25T01:22:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.985+0000 7fbb94e37640 1 -- start start 2024-04-25T01:22:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb94e37640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbb900b37c0 0x7fbb900cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb94e37640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb90158a70 0x7fbb9013b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb94e37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb9015bc30 0x7fbb900aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb94e37640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9006cb70 con 0x7fbb9015bc30 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb94e37640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9006c9f0 con 0x7fbb90158a70 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb94e37640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbb9006ccf0 con 0x7fbb900b37c0 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb8ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb9015bc30 0x7fbb900aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb8ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb9015bc30 0x7fbb900aff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48156/0 (socket says 172.21.15.77:48156) 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb8ed9d640 1 -- 172.21.15.77:0/3342032825 learned_addr learned my addr 172.21.15.77:0/3342032825 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb8dd9b640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb90158a70 0x7fbb9013b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb8e59c640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbb900b37c0 0x7fbb900cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb8ed9d640 1 -- 172.21.15.77:0/3342032825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbb900b37c0 msgr2=0x7fbb900cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb8ed9d640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbb900b37c0 0x7fbb900cdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb8ed9d640 1 -- 172.21.15.77:0/3342032825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb90158a70 msgr2=0x7fbb9013b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb8ed9d640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb90158a70 0x7fbb9013b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb8ed9d640 1 -- 172.21.15.77:0/3342032825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb900ce860 con 0x7fbb9015bc30 2024-04-25T01:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.986+0000 7fbb8ed9d640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb9015bc30 0x7fbb900aff10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800b440 tx=0x7fbb8800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.987+0000 7fbb7effd640 1 -- 172.21.15.77:0/3342032825 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb88015020 con 0x7fbb9015bc30 2024-04-25T01:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.987+0000 7fbb7effd640 1 -- 172.21.15.77:0/3342032825 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb88011830 con 0x7fbb9015bc30 2024-04-25T01:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.987+0000 7fbb94e37640 1 -- 172.21.15.77:0/3342032825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb900ceb40 con 0x7fbb9015bc30 2024-04-25T01:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.987+0000 7fbb7effd640 1 -- 172.21.15.77:0/3342032825 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb88011c20 con 0x7fbb9015bc30 2024-04-25T01:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.987+0000 7fbb94e37640 1 -- 172.21.15.77:0/3342032825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb900cf100 con 0x7fbb9015bc30 2024-04-25T01:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.988+0000 7fbb94e37640 1 -- 172.21.15.77:0/3342032825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb90068fb0 con 0x7fbb9015bc30 2024-04-25T01:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.988+0000 7fbb7effd640 1 -- 172.21.15.77:0/3342032825 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbb88018020 con 0x7fbb9015bc30 2024-04-25T01:22:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.988+0000 7fbb7effd640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbb7006be30 0x7fbb7006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.988+0000 7fbb7effd640 1 -- 172.21.15.77:0/3342032825 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fbb8808b160 con 0x7fbb9015bc30 2024-04-25T01:22:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.988+0000 7fbb8e59c640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbb7006be30 0x7fbb7006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.989+0000 7fbb8e59c640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbb7006be30 0x7fbb7006e2f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbb80033000 tx=0x7fbb80009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:41.993+0000 7fbb7effd640 1 -- 172.21.15.77:0/3342032825 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbb8801f020 con 0x7fbb9015bc30 2024-04-25T01:22:42.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.169+0000 7fbb94e37640 1 -- 172.21.15.77:0/3342032825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbb900b3c70 con 0x7fbb9015bc30 2024-04-25T01:22:42.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.170+0000 7fbb7effd640 1 -- 172.21.15.77:0/3342032825 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fbb880172c0 con 0x7fbb9015bc30 2024-04-25T01:22:42.151 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.171+0000 7fbb94e37640 1 -- 172.21.15.77:0/3342032825 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbb7006be30 msgr2=0x7fbb7006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.171+0000 7fbb94e37640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbb7006be30 0x7fbb7006e2f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbb80033000 tx=0x7fbb80009000 comp rx=0 tx=0).stop 2024-04-25T01:22:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.171+0000 7fbb94e37640 1 -- 172.21.15.77:0/3342032825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb9015bc30 msgr2=0x7fbb900aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.171+0000 7fbb94e37640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb9015bc30 0x7fbb900aff10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800b440 tx=0x7fbb8800b900 comp rx=0 tx=0).stop 2024-04-25T01:22:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.171+0000 7fbb94e37640 1 -- 172.21.15.77:0/3342032825 shutdown_connections 2024-04-25T01:22:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.171+0000 7fbb94e37640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbb7006be30 0x7fbb7006e2f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.171+0000 7fbb94e37640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb9015bc30 0x7fbb900aff10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.171+0000 7fbb94e37640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb90158a70 0x7fbb9013b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.171+0000 7fbb94e37640 1 --2- 172.21.15.77:0/3342032825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbb900b37c0 0x7fbb900cdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.171+0000 7fbb94e37640 1 -- 172.21.15.77:0/3342032825 >> 172.21.15.77:0/3342032825 conn(0x7fbb9006b720 msgr2=0x7fbb90070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.172+0000 7fbb94e37640 1 -- 172.21.15.77:0/3342032825 shutdown_connections 2024-04-25T01:22:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.172+0000 7fbb94e37640 1 -- 172.21.15.77:0/3342032825 wait complete. 2024-04-25T01:22:42.164 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:42.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.282+0000 7fe6f3d96640 1 Processor -- start 2024-04-25T01:22:42.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.282+0000 7fe6f3d96640 1 -- start start 2024-04-25T01:22:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.283+0000 7fe6f3d96640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6ec0722e0 0x7fe6ec06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.283+0000 7fe6f3d96640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ec06e410 con 0x7fe6ec15fc40 2024-04-25T01:22:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.283+0000 7fe6f3d96640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ec06d110 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.283+0000 7fe6f3d96640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fe6ec160aa0 con 0x7fe6ec0722e0 2024-04-25T01:22:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.283+0000 7fe6f1b32640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6ec0722e0 0x7fe6ec06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.283+0000 7fe6f1331640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe6ec15fc40 0x7fe6ec0cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33072/0 (socket says 172.21.15.77:33072) 2024-04-25T01:22:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.283+0000 7fe6f1331640 1 -- 172.21.15.77:0/1425952977 learned_addr learned my addr 172.21.15.77:0/1425952977 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.283+0000 7fe6f1b32640 1 -- 172.21.15.77:0/1425952977 >> v1:172.21.15.121:6789/0 conn(0x7fe6ec0cf4d0 legacy=0x7fe6ec070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.283+0000 7fe6f1b32640 1 -- 172.21.15.77:0/1425952977 >> v1:172.21.15.77:6789/0 conn(0x7fe6ec15fc40 legacy=0x7fe6ec0cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.283+0000 7fe6f1b32640 1 -- 172.21.15.77:0/1425952977 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ec06d3f0 con 0x7fe6ec0722e0 2024-04-25T01:22:42.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.283+0000 7fe6f0b30640 1 -- 172.21.15.77:0/1425952977 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 554691137 0 0) 0x7fe6ec06e410 con 0x7fe6ec15fc40 2024-04-25T01:22:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.284+0000 7fe6f1b32640 1 --2- 172.21.15.77:0/1425952977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ec0722e0 0x7fe6ec06ded0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009d90 tx=0x7fe6e802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75771059e203b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.284+0000 7fe6f0b30640 1 -- 172.21.15.77:0/1425952977 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8031600 con 0x7fe6ec0722e0 2024-04-25T01:22:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.284+0000 7fe6f0b30640 1 -- 172.21.15.77:0/1425952977 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e80317a0 con 0x7fe6ec0722e0 2024-04-25T01:22:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.284+0000 7fe6f0b30640 1 -- 172.21.15.77:0/1425952977 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8031b70 con 0x7fe6ec0722e0 2024-04-25T01:22:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.285+0000 7fe6f3d96640 1 -- 172.21.15.77:0/1425952977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ec0722e0 msgr2=0x7fe6ec06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.285+0000 7fe6f3d96640 1 --2- 172.21.15.77:0/1425952977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ec0722e0 0x7fe6ec06ded0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009d90 tx=0x7fe6e802a610 comp rx=0 tx=0).stop 2024-04-25T01:22:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.285+0000 7fe6f3d96640 1 -- 172.21.15.77:0/1425952977 shutdown_connections 2024-04-25T01:22:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.285+0000 7fe6f3d96640 1 --2- 172.21.15.77:0/1425952977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ec0722e0 0x7fe6ec06ded0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.285+0000 7fe6f3d96640 1 -- 172.21.15.77:0/1425952977 >> 172.21.15.77:0/1425952977 conn(0x7fe6ec06b720 msgr2=0x7fe6ec0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.285+0000 7fe6f3d96640 1 -- 172.21.15.77:0/1425952977 shutdown_connections 2024-04-25T01:22:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.285+0000 7fe6f3d96640 1 -- 172.21.15.77:0/1425952977 wait complete. 2024-04-25T01:22:42.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.286+0000 7fe6f3d96640 1 Processor -- start 2024-04-25T01:22:42.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.286+0000 7fe6f3d96640 1 -- start start 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f3d96640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6ec0722e0 0x7fe6ec155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f3d96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ec0cf4d0 0x7fe6ec156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f3d96640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec15fc40 0x7fe6ec132270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f3d96640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec06cec0 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f3d96640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec071260 con 0x7fe6ec0722e0 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f3d96640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe6ec0713e0 con 0x7fe6ec15fc40 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f1331640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ec0cf4d0 0x7fe6ec156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f1331640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ec0cf4d0 0x7fe6ec156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48170/0 (socket says 172.21.15.77:48170) 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f1331640 1 -- 172.21.15.77:0/139583400 learned_addr learned my addr 172.21.15.77:0/139583400 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f1331640 1 -- 172.21.15.77:0/139583400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec15fc40 msgr2=0x7fe6ec132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f2333640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec15fc40 0x7fe6ec132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f1b32640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6ec0722e0 0x7fe6ec155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f1331640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec15fc40 0x7fe6ec132270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f1331640 1 -- 172.21.15.77:0/139583400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6ec0722e0 msgr2=0x7fe6ec155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f1331640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6ec0722e0 0x7fe6ec155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f1331640 1 -- 172.21.15.77:0/139583400 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ec157560 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f1331640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ec0cf4d0 0x7fe6ec156d10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00c430 tx=0x7fe6dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6e27fc640 1 -- 172.21.15.77:0/139583400 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc0074e0 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6e27fc640 1 -- 172.21.15.77:0/139583400 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6dc0047e0 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6f3d96640 1 -- 172.21.15.77:0/139583400 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ec15e3a0 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.287+0000 7fe6e27fc640 1 -- 172.21.15.77:0/139583400 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc004bd0 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.288+0000 7fe6f3d96640 1 -- 172.21.15.77:0/139583400 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ec15e960 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.289+0000 7fe6f3d96640 1 -- 172.21.15.77:0/139583400 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6ec068fb0 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.289+0000 7fe6e27fc640 1 -- 172.21.15.77:0/139583400 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe6dc011070 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.289+0000 7fe6e27fc640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe6d006be60 0x7fe6d006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.289+0000 7fe6f1b32640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe6d006be60 0x7fe6d006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.289+0000 7fe6e27fc640 1 -- 172.21.15.77:0/139583400 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fe6dc08ae10 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.290+0000 7fe6f1b32640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe6d006be60 0x7fe6d006e320 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8034000 tx=0x7fe6e802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.294+0000 7fe6e27fc640 1 -- 172.21.15.77:0/139583400 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe6dc00e280 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.468+0000 7fe6f3d96640 1 -- 172.21.15.77:0/139583400 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe6ec072c90 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.469+0000 7fe6e27fc640 1 -- 172.21.15.77:0/139583400 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fe6dc059440 con 0x7fe6ec0cf4d0 2024-04-25T01:22:42.450 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:42.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.470+0000 7fe6f3d96640 1 -- 172.21.15.77:0/139583400 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe6d006be60 msgr2=0x7fe6d006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.470+0000 7fe6f3d96640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe6d006be60 0x7fe6d006e320 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8034000 tx=0x7fe6e802f000 comp rx=0 tx=0).stop 2024-04-25T01:22:42.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.470+0000 7fe6f3d96640 1 -- 172.21.15.77:0/139583400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ec0cf4d0 msgr2=0x7fe6ec156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.470+0000 7fe6f3d96640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ec0cf4d0 0x7fe6ec156d10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00c430 tx=0x7fe6dc00c8f0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.471+0000 7fe6f3d96640 1 -- 172.21.15.77:0/139583400 shutdown_connections 2024-04-25T01:22:42.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.471+0000 7fe6f3d96640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe6d006be60 0x7fe6d006e320 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.471+0000 7fe6f3d96640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ec0cf4d0 0x7fe6ec156d10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.471+0000 7fe6f3d96640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec15fc40 0x7fe6ec132270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.471+0000 7fe6f3d96640 1 --2- 172.21.15.77:0/139583400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6ec0722e0 0x7fe6ec155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.471+0000 7fe6f3d96640 1 -- 172.21.15.77:0/139583400 >> 172.21.15.77:0/139583400 conn(0x7fe6ec06b720 msgr2=0x7fe6ec15cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:42.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.471+0000 7fe6f3d96640 1 -- 172.21.15.77:0/139583400 shutdown_connections 2024-04-25T01:22:42.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.471+0000 7fe6f3d96640 1 -- 172.21.15.77:0/139583400 wait complete. 2024-04-25T01:22:42.463 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:22:42.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.580+0000 7f4d0f9d5640 1 Processor -- start 2024-04-25T01:22:42.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.580+0000 7f4d0f9d5640 1 -- start start 2024-04-25T01:22:42.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.581+0000 7f4d0f9d5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4d08161aa0 0x7f4d080b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.581+0000 7f4d0f9d5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d080d1120 con 0x7f4d080d2880 2024-04-25T01:22:42.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.581+0000 7f4d0f9d5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d080d1400 con 0x7f4d080d24a0 2024-04-25T01:22:42.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.581+0000 7f4d0f9d5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f4d080d16e0 con 0x7f4d08161aa0 2024-04-25T01:22:42.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.581+0000 7f4d0df72640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4d08161aa0 0x7f4d080b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.581+0000 7f4d0cf70640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4d080d2880 0x7f4d08161390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33078/0 (socket says 172.21.15.77:33078) 2024-04-25T01:22:42.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.581+0000 7f4d0cf70640 1 -- 172.21.15.77:0/3960204405 learned_addr learned my addr 172.21.15.77:0/3960204405 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:42.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.581+0000 7f4d0df72640 1 -- 172.21.15.77:0/3960204405 >> v1:172.21.15.121:6789/0 conn(0x7f4d080d24a0 legacy=0x7f4d080d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:42.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.581+0000 7f4d0df72640 1 -- 172.21.15.77:0/3960204405 >> v1:172.21.15.77:6789/0 conn(0x7f4d080d2880 legacy=0x7f4d08161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.581+0000 7f4d0df72640 1 -- 172.21.15.77:0/3960204405 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d080cee60 con 0x7f4d08161aa0 2024-04-25T01:22:42.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.581+0000 7f4cfffff640 1 -- 172.21.15.77:0/3960204405 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 305128167 0 0) 0x7f4d080d1120 con 0x7f4d080d2880 2024-04-25T01:22:42.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.582+0000 7f4d0df72640 1 --2- 172.21.15.77:0/3960204405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d08161aa0 0x7f4d080b0bd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009d90 tx=0x7f4d0002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c3693ce15a6cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.582+0000 7f4cfffff640 1 -- 172.21.15.77:0/3960204405 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00007740 con 0x7f4d08161aa0 2024-04-25T01:22:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.582+0000 7f4cfffff640 1 -- 172.21.15.77:0/3960204405 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d000078e0 con 0x7f4d08161aa0 2024-04-25T01:22:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.582+0000 7f4cfffff640 1 -- 172.21.15.77:0/3960204405 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00007cb0 con 0x7f4d08161aa0 2024-04-25T01:22:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.582+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/3960204405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d08161aa0 msgr2=0x7f4d080b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.582+0000 7f4d0f9d5640 1 --2- 172.21.15.77:0/3960204405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d08161aa0 0x7f4d080b0bd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009d90 tx=0x7f4d0002a6a0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.583+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/3960204405 shutdown_connections 2024-04-25T01:22:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.583+0000 7f4d0f9d5640 1 --2- 172.21.15.77:0/3960204405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d08161aa0 0x7f4d080b0bd0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.583+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/3960204405 >> 172.21.15.77:0/3960204405 conn(0x7f4d0809d200 msgr2=0x7f4d080d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.583+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/3960204405 shutdown_connections 2024-04-25T01:22:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.583+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/3960204405 wait complete. 2024-04-25T01:22:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.584+0000 7f4d0f9d5640 1 Processor -- start 2024-04-25T01:22:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.584+0000 7f4d0f9d5640 1 -- start start 2024-04-25T01:22:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.584+0000 7f4d0f9d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d080d24a0 0x7f4d08158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.584+0000 7f4d0f9d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d080d2880 0x7f4d081f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.584+0000 7f4d0f9d5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08161aa0 0x7f4d0815fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.584+0000 7f4d0f9d5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4d080b2f20 con 0x7f4d080d2880 2024-04-25T01:22:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.584+0000 7f4d0f9d5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4d080b2da0 con 0x7f4d08161aa0 2024-04-25T01:22:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.584+0000 7f4d0f9d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4d080b30a0 con 0x7f4d080d24a0 2024-04-25T01:22:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4d0cf70640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d080d2880 0x7f4d081f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4d0cf70640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d080d2880 0x7f4d081f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48184/0 (socket says 172.21.15.77:48184) 2024-04-25T01:22:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4d0cf70640 1 -- 172.21.15.77:0/282176975 learned_addr learned my addr 172.21.15.77:0/282176975 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4d0cf70640 1 -- 172.21.15.77:0/282176975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d080d24a0 msgr2=0x7f4d08158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4d0cf70640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d080d24a0 0x7f4d08158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4d0cf70640 1 -- 172.21.15.77:0/282176975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08161aa0 msgr2=0x7f4d0815fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4d0cf70640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08161aa0 0x7f4d0815fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4d0cf70640 1 -- 172.21.15.77:0/282176975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d081f4ec0 con 0x7f4d080d2880 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4d0cf70640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d080d2880 0x7f4d081f43b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800c430 tx=0x7f4cf800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4cfdffb640 1 -- 172.21.15.77:0/282176975 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf80074e0 con 0x7f4d080d2880 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4cfdffb640 1 -- 172.21.15.77:0/282176975 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cf80047e0 con 0x7f4d080d2880 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/282176975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d081f51a0 con 0x7f4d080d2880 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4cfdffb640 1 -- 172.21.15.77:0/282176975 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8004bd0 con 0x7f4d080d2880 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.585+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/282176975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d080c9420 con 0x7f4d080d2880 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.586+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/282176975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d08068fb0 con 0x7f4d080d2880 2024-04-25T01:22:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.587+0000 7f4cfdffb640 1 -- 172.21.15.77:0/282176975 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4cf8011070 con 0x7f4d080d2880 2024-04-25T01:22:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.587+0000 7f4cfdffb640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4cec06be30 0x7f4cec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.587+0000 7f4cfdffb640 1 -- 172.21.15.77:0/282176975 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f4cf808af10 con 0x7f4d080d2880 2024-04-25T01:22:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.587+0000 7f4d0d771640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4cec06be30 0x7f4cec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.587+0000 7f4d0d771640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4cec06be30 0x7f4cec06e2f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d080d4bc0 tx=0x7f4d04006ba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.591+0000 7f4cfdffb640 1 -- 172.21.15.77:0/282176975 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4cf80594b0 con 0x7f4d080d2880 2024-04-25T01:22:42.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.777+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/282176975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4d0815f250 con 0x7f4d080d2880 2024-04-25T01:22:42.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.777+0000 7f4cfdffb640 1 -- 172.21.15.77:0/282176975 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f4cf8058db0 con 0x7f4d080d2880 2024-04-25T01:22:42.759 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-04-25T01:22:42.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.779+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/282176975 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4cec06be30 msgr2=0x7f4cec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.779+0000 7f4d0f9d5640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4cec06be30 0x7f4cec06e2f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d080d4bc0 tx=0x7f4d04006ba0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.779+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/282176975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d080d2880 msgr2=0x7f4d081f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.779+0000 7f4d0f9d5640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d080d2880 0x7f4d081f43b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800c430 tx=0x7f4cf800c8f0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.780+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/282176975 shutdown_connections 2024-04-25T01:22:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.780+0000 7f4d0f9d5640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4cec06be30 0x7f4cec06e2f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.780+0000 7f4d0f9d5640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d080d2880 0x7f4d081f43b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.780+0000 7f4d0f9d5640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08161aa0 0x7f4d0815fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.780+0000 7f4d0f9d5640 1 --2- 172.21.15.77:0/282176975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d080d24a0 0x7f4d08158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.780+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/282176975 >> 172.21.15.77:0/282176975 conn(0x7f4d0809d200 msgr2=0x7f4d080d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.780+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/282176975 shutdown_connections 2024-04-25T01:22:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.780+0000 7f4d0f9d5640 1 -- 172.21.15.77:0/282176975 wait complete. 2024-04-25T01:22:42.772 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-04-25T01:22:42.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:42.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-04-25T01:22:42.802 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:22:42.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:22:42.803 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-25T01:22:42.862 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T01:22:42.862 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:42.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.977+0000 7f1ce77a5640 1 Processor -- start 2024-04-25T01:22:42.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.978+0000 7f1ce77a5640 1 -- start start 2024-04-25T01:22:42.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.978+0000 7f1ce77a5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1ce00cfaf0 0x7f1ce00d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.978+0000 7f1ce77a5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ce00d1e90 con 0x7f1ce015e4c0 2024-04-25T01:22:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.978+0000 7f1ce77a5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ce00b54a0 con 0x7f1ce00d4a10 2024-04-25T01:22:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.978+0000 7f1ce77a5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1ce00d2130 con 0x7f1ce00cfaf0 2024-04-25T01:22:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.978+0000 7f1ce5541640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1ce015e4c0 0x7f1ce00d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33086/0 (socket says 172.21.15.77:33086) 2024-04-25T01:22:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1ce5541640 1 -- 172.21.15.77:0/1748166010 learned_addr learned my addr 172.21.15.77:0/1748166010 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1ce5d42640 1 --2- 172.21.15.77:0/1748166010 >> v2:172.21.15.121:3300/0 conn(0x7f1ce00cfaf0 0x7f1ce00d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2127968612 0 0) 0x7f1ce00d1e90 con 0x7f1ce015e4c0 2024-04-25T01:22:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cc8003520 con 0x7f1ce015e4c0 2024-04-25T01:22:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3568810948 0 0) 0x7f1cc8003520 con 0x7f1ce015e4c0 2024-04-25T01:22:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 >> v2:172.21.15.121:3300/0 conn(0x7f1ce00cfaf0 msgr2=0x7f1ce00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 --2- 172.21.15.77:0/1748166010 >> v2:172.21.15.121:3300/0 conn(0x7f1ce00cfaf0 0x7f1ce00d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 >> v1:172.21.15.121:6789/0 conn(0x7f1ce00d4a10 legacy=0x7f1ce00b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce00b5780 con 0x7f1ce015e4c0 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f1cd80019f0 con 0x7f1ce015e4c0 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 >> v1:172.21.15.77:6789/0 conn(0x7f1ce015e4c0 legacy=0x7f1ce00d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80039b0 0x7f1cc8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc8004800 0x7f1cc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc80051c0 0x7f1cc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce00d2130 con 0x7f1cc8004800 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc800c160 con 0x7f1cc80039b0 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1cc800c360 con 0x7f1cc80051c0 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1ce4d40640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc8004800 0x7f1cc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1ce5541640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80039b0 0x7f1cc8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1ce5d42640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc80051c0 0x7f1cc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1ce4d40640 1 -- 172.21.15.77:0/1748166010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc80051c0 msgr2=0x7f1cc800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1ce4d40640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc80051c0 0x7f1cc800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1ce4d40640 1 -- 172.21.15.77:0/1748166010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80039b0 msgr2=0x7f1cc8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1ce4d40640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80039b0 0x7f1cc8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.979+0000 7f1ce4d40640 1 -- 172.21.15.77:0/1748166010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc8003520 con 0x7f1cc8004800 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.980+0000 7f1ce5d42640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cc80051c0 0x7f1cc800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.980+0000 7f1ce5d42640 1 -- 172.21.15.77:0/1748166010 reap_dead start 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.980+0000 7f1ce4d40640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc8004800 0x7f1cc8004c80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4001720 tx=0x7f1cc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec674130ac386e25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.980+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4001370 con 0x7f1cc8004800 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.980+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cd40052d0 con 0x7f1cc8004800 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.980+0000 7f1cdffff640 1 -- 172.21.15.77:0/1748166010 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd402ab90 con 0x7f1cc8004800 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.981+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1748166010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc8004800 msgr2=0x7f1cc8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.981+0000 7f1ce77a5640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc8004800 0x7f1cc8004c80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4001720 tx=0x7f1cc8005830 comp rx=0 tx=0).stop 2024-04-25T01:22:42.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.981+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1748166010 shutdown_connections 2024-04-25T01:22:42.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.981+0000 7f1ce77a5640 1 --2- 172.21.15.77:0/1748166010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc8004800 0x7f1cc8004c80 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.981+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1748166010 >> 172.21.15.77:0/1748166010 conn(0x7f1ce00d3ef0 msgr2=0x7f1ce00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:42.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.981+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1748166010 shutdown_connections 2024-04-25T01:22:42.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.981+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1748166010 wait complete. 2024-04-25T01:22:42.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.982+0000 7f1ce77a5640 1 Processor -- start 2024-04-25T01:22:42.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.982+0000 7f1ce77a5640 1 -- start start 2024-04-25T01:22:42.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce77a5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc8004800 0x7f1ce01f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce77a5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce0159a00 0x7f1ce0152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce77a5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ce0159e80 0x7f1ce0153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce77a5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce015ea20 con 0x7f1ce0159a00 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce77a5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce015e8a0 con 0x7f1cc8004800 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce77a5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce00d2800 con 0x7f1ce0159e80 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce4d40640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce0159a00 0x7f1ce0152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce4d40640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce0159a00 0x7f1ce0152fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48194/0 (socket says 172.21.15.77:48194) 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce4d40640 1 -- 172.21.15.77:0/1557451853 learned_addr learned my addr 172.21.15.77:0/1557451853 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce5541640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc8004800 0x7f1ce01f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce5d42640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ce0159e80 0x7f1ce0153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce4d40640 1 -- 172.21.15.77:0/1557451853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ce0159e80 msgr2=0x7f1ce0153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce4d40640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ce0159e80 0x7f1ce0153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce4d40640 1 -- 172.21.15.77:0/1557451853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc8004800 msgr2=0x7f1ce01f4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce4d40640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc8004800 0x7f1ce01f4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce4d40640 1 -- 172.21.15.77:0/1557451853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce00cc430 con 0x7f1ce0159a00 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce5d42640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ce0159e80 0x7f1ce0153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce4d40640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce0159a00 0x7f1ce0152fc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4001840 tx=0x7f1cd4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1cddffb640 1 -- 172.21.15.77:0/1557451853 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd403f030 con 0x7f1ce0159a00 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1cddffb640 1 -- 172.21.15.77:0/1557451853 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cd4005480 con 0x7f1ce0159a00 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1557451853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce00cc710 con 0x7f1ce0159a00 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.983+0000 7f1cddffb640 1 -- 172.21.15.77:0/1557451853 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4037040 con 0x7f1ce0159a00 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.984+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1557451853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce00cccd0 con 0x7f1ce0159a00 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.984+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1557451853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ce0068fb0 con 0x7f1ce0159a00 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.985+0000 7f1cddffb640 1 -- 172.21.15.77:0/1557451853 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1cd4042070 con 0x7f1ce0159a00 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.985+0000 7f1cddffb640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ccc06be30 0x7f1ccc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.985+0000 7f1cddffb640 1 -- 172.21.15.77:0/1557451853 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f1cd40aacb0 con 0x7f1ce0159a00 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.985+0000 7f1ce5541640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ccc06be30 0x7f1ccc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:42.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.986+0000 7f1ce5541640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ccc06be30 0x7f1ccc06e2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8002230 tx=0x7f1cd8006040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:42.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:42.990+0000 7f1cddffb640 1 -- 172.21.15.77:0/1557451853 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1cd4079250 con 0x7f1ce0159a00 2024-04-25T01:22:43.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.200+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1557451853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ce006c370 con 0x7f1ce0159a00 2024-04-25T01:22:43.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.201+0000 7f1cddffb640 1 -- 172.21.15.77:0/1557451853 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7f1cd4078b50 con 0x7f1ce0159a00 2024-04-25T01:22:43.183 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:43.183 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4267907621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4267907621},{"type":"v1","addr":"172.21.15.77:6837","nonce":4267907621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/493518847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":493518847},{"type":"v1","addr":"172.21.15.121:6833","nonce":493518847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:21:45.408154+0000","modified":"2024-04-25T01:22:06.554124+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6835/2810316639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2810316639},{"type":"v1","addr":"172.21.15.121:6835","nonce":2810316639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T01:22:43.183 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2024-04-25T01:22:43.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1557451853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ccc06be30 msgr2=0x7f1ccc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ccc06be30 0x7f1ccc06e2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8002230 tx=0x7f1cd8006040 comp rx=0 tx=0).stop 2024-04-25T01:22:43.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1557451853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce0159a00 msgr2=0x7f1ce0152fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce0159a00 0x7f1ce0152fc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4001840 tx=0x7f1cd4001120 comp rx=0 tx=0).stop 2024-04-25T01:22:43.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1557451853 shutdown_connections 2024-04-25T01:22:43.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1ccc06be30 0x7f1ccc06e2f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce0159a00 0x7f1ce0152fc0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ce0159e80 0x7f1ce0153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 --2- 172.21.15.77:0/1557451853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc8004800 0x7f1ce01f4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1557451853 >> 172.21.15.77:0/1557451853 conn(0x7f1ce00d3ef0 msgr2=0x7f1ce00b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:43.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1557451853 shutdown_connections 2024-04-25T01:22:43.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.203+0000 7f1ce77a5640 1 -- 172.21.15.77:0/1557451853 wait complete. 2024-04-25T01:22:43.196 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:43.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.311+0000 7f64e651b640 1 Processor -- start 2024-04-25T01:22:43.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.311+0000 7f64e651b640 1 -- start start 2024-04-25T01:22:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64e651b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64e0143940 0x7f64e006e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64e651b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64e0143d20 0x7f64e006eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64e651b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64e0070e60 con 0x7f64e015ecf0 2024-04-25T01:22:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64e651b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f64e0071140 con 0x7f64e0143d20 2024-04-25T01:22:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64e651b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64e0071420 con 0x7f64e0143940 2024-04-25T01:22:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64df7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64e0143d20 0x7f64e006eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64e4ab8640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f64e015ecf0 0x7f64e015f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33088/0 (socket says 172.21.15.77:33088) 2024-04-25T01:22:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64e4ab8640 1 -- 172.21.15.77:0/3639876655 learned_addr learned my addr 172.21.15.77:0/3639876655 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64df7fe640 1 -- 172.21.15.77:0/3639876655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64e0143940 msgr2=0x7f64e006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64df7fe640 1 --2- 172.21.15.77:0/3639876655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64e0143940 0x7f64e006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64df7fe640 1 -- 172.21.15.77:0/3639876655 >> v1:172.21.15.77:6789/0 conn(0x7f64e015ecf0 legacy=0x7f64e015f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64df7fe640 1 -- 172.21.15.77:0/3639876655 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e0071700 con 0x7f64e0143d20 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.312+0000 7f64deffd640 1 -- 172.21.15.77:0/3639876655 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4225908813 0 0) 0x7f64e0070e60 con 0x7f64e015ecf0 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.313+0000 7f64df7fe640 1 --2- 172.21.15.77:0/3639876655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e0143d20 0x7f64e006eb90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f64d40097a0 tx=0x7f64d402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8548e80155cde78c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.313+0000 7f64deffd640 1 -- 172.21.15.77:0/3639876655 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d4037070 con 0x7f64e0143d20 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.313+0000 7f64deffd640 1 -- 172.21.15.77:0/3639876655 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64d4032440 con 0x7f64e0143d20 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.313+0000 7f64deffd640 1 -- 172.21.15.77:0/3639876655 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d4032810 con 0x7f64e0143d20 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.313+0000 7f64e651b640 1 -- 172.21.15.77:0/3639876655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e0143d20 msgr2=0x7f64e006eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.313+0000 7f64e651b640 1 --2- 172.21.15.77:0/3639876655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e0143d20 0x7f64e006eb90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f64d40097a0 tx=0x7f64d402a180 comp rx=0 tx=0).stop 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.313+0000 7f64e651b640 1 -- 172.21.15.77:0/3639876655 shutdown_connections 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.313+0000 7f64e651b640 1 --2- 172.21.15.77:0/3639876655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e0143d20 0x7f64e006eb90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.313+0000 7f64e651b640 1 --2- 172.21.15.77:0/3639876655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64e0143940 0x7f64e006e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.313+0000 7f64e651b640 1 -- 172.21.15.77:0/3639876655 >> 172.21.15.77:0/3639876655 conn(0x7f64e00b1820 msgr2=0x7f64e00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:43.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.313+0000 7f64e651b640 1 -- 172.21.15.77:0/3639876655 shutdown_connections 2024-04-25T01:22:43.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.314+0000 7f64e651b640 1 -- 172.21.15.77:0/3639876655 wait complete. 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.315+0000 7f64e651b640 1 Processor -- start 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.315+0000 7f64e651b640 1 -- start start 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.315+0000 7f64e651b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64e0143940 0x7f64e0156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.315+0000 7f64e651b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e0143d20 0x7f64e00ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.315+0000 7f64e651b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64e015ecf0 0x7f64e013f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.315+0000 7f64e651b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f64e0071fb0 con 0x7f64e0143d20 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.315+0000 7f64e651b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f64e0071e30 con 0x7f64e0143940 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.315+0000 7f64e651b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64e0072130 con 0x7f64e015ecf0 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64df7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e0143d20 0x7f64e00ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64df7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e0143d20 0x7f64e00ce840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48212/0 (socket says 172.21.15.77:48212) 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64df7fe640 1 -- 172.21.15.77:0/1254540491 learned_addr learned my addr 172.21.15.77:0/1254540491 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64dffff640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64e0143940 0x7f64e0156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64df7fe640 1 -- 172.21.15.77:0/1254540491 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64e015ecf0 msgr2=0x7f64e013f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64df7fe640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64e015ecf0 0x7f64e013f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64df7fe640 1 -- 172.21.15.77:0/1254540491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64e0143940 msgr2=0x7f64e0156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64df7fe640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64e0143940 0x7f64e0156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64df7fe640 1 -- 172.21.15.77:0/1254540491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e0156810 con 0x7f64e0143d20 2024-04-25T01:22:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64dffff640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64e0143940 0x7f64e0156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64df7fe640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e0143d20 0x7f64e00ce840 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f64d40098c0 tx=0x7f64d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64dcff9640 1 -- 172.21.15.77:0/1254540491 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d4041070 con 0x7f64e0143d20 2024-04-25T01:22:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64dcff9640 1 -- 172.21.15.77:0/1254540491 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64d403c870 con 0x7f64e0143d20 2024-04-25T01:22:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64e651b640 1 -- 172.21.15.77:0/1254540491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64e0156af0 con 0x7f64e0143d20 2024-04-25T01:22:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64dcff9640 1 -- 172.21.15.77:0/1254540491 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d4037040 con 0x7f64e0143d20 2024-04-25T01:22:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.316+0000 7f64e651b640 1 -- 172.21.15.77:0/1254540491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64e01570b0 con 0x7f64e0143d20 2024-04-25T01:22:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.317+0000 7f64e651b640 1 -- 172.21.15.77:0/1254540491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64e00b3450 con 0x7f64e0143d20 2024-04-25T01:22:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.318+0000 7f64dcff9640 1 -- 172.21.15.77:0/1254540491 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f64d4044070 con 0x7f64e0143d20 2024-04-25T01:22:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.318+0000 7f64dcff9640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64cc06be30 0x7f64cc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.318+0000 7f64dcff9640 1 -- 172.21.15.77:0/1254540491 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f64d40abd20 con 0x7f64e0143d20 2024-04-25T01:22:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.318+0000 7f64dffff640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64cc06be30 0x7f64cc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.319+0000 7f64dffff640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64cc06be30 0x7f64cc06e2f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f64d0006fd0 tx=0x7f64d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.322+0000 7f64dcff9640 1 -- 172.21.15.77:0/1254540491 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f64e00b3450 con 0x7f64e0143d20 2024-04-25T01:22:43.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.500+0000 7f64e651b640 1 -- 172.21.15.77:0/1254540491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64e00ced80 con 0x7f64e0143d20 2024-04-25T01:22:43.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.501+0000 7f64dcff9640 1 -- 172.21.15.77:0/1254540491 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18922 (secure 0 0 0) 0x7f64e00ced80 con 0x7f64e0143d20 2024-04-25T01:22:43.483 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:43.483 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:26.287128+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.503+0000 7f64e651b640 1 -- 172.21.15.77:0/1254540491 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64cc06be30 msgr2=0x7f64cc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.503+0000 7f64e651b640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64cc06be30 0x7f64cc06e2f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f64d0006fd0 tx=0x7f64d0008040 comp rx=0 tx=0).stop 2024-04-25T01:22:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.503+0000 7f64e651b640 1 -- 172.21.15.77:0/1254540491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e0143d20 msgr2=0x7f64e00ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.503+0000 7f64e651b640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e0143d20 0x7f64e00ce840 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f64d40098c0 tx=0x7f64d40046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.503+0000 7f64e651b640 1 -- 172.21.15.77:0/1254540491 shutdown_connections 2024-04-25T01:22:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.503+0000 7f64e651b640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f64cc06be30 0x7f64cc06e2f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.503+0000 7f64e651b640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e0143d20 0x7f64e00ce840 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.503+0000 7f64e651b640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64e015ecf0 0x7f64e013f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.503+0000 7f64e651b640 1 --2- 172.21.15.77:0/1254540491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64e0143940 0x7f64e0156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.503+0000 7f64e651b640 1 -- 172.21.15.77:0/1254540491 >> 172.21.15.77:0/1254540491 conn(0x7f64e00b1820 msgr2=0x7f64e00b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:43.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.504+0000 7f64e651b640 1 -- 172.21.15.77:0/1254540491 shutdown_connections 2024-04-25T01:22:43.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.504+0000 7f64e651b640 1 -- 172.21.15.77:0/1254540491 wait complete. 2024-04-25T01:22:43.496 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:21:43.207433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:21:44.385674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:21:48.116140+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:22:43.497 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.615+0000 7fcc67888640 1 Processor -- start 2024-04-25T01:22:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.615+0000 7fcc67888640 1 -- start start 2024-04-25T01:22:43.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc67888640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcc600b47d0 0x7fcc6015b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc67888640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc600b37c0 0x7fcc6015ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc67888640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc6015bfa0 con 0x7fcc60158a70 2024-04-25T01:22:43.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc67888640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcc6015c210 con 0x7fcc600b47d0 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc67888640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcc600b2ce0 con 0x7fcc600b37c0 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc65624640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcc60158a70 0x7fcc600b40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33094/0 (socket says 172.21.15.77:33094) 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc65624640 1 -- 172.21.15.77:0/66307547 learned_addr learned my addr 172.21.15.77:0/66307547 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc64e23640 1 --2- 172.21.15.77:0/66307547 >> v2:172.21.15.121:3300/0 conn(0x7fcc600b47d0 0x7fcc6015b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3524301530 0 0) 0x7fcc6015bfa0 con 0x7fcc60158a70 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc65e25640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc600b37c0 0x7fcc6015ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc4c003520 con 0x7fcc60158a70 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 933839704 0 0) 0x7fcc4c003520 con 0x7fcc60158a70 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc600b37c0 msgr2=0x7fcc6015ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc600b37c0 0x7fcc6015ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 >> v2:172.21.15.121:3300/0 conn(0x7fcc600b47d0 msgr2=0x7fcc6015b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 --2- 172.21.15.77:0/66307547 >> v2:172.21.15.121:3300/0 conn(0x7fcc600b47d0 0x7fcc6015b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc600b2fc0 con 0x7fcc60158a70 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fcc5c001670 con 0x7fcc60158a70 2024-04-25T01:22:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 >> v1:172.21.15.77:6789/0 conn(0x7fcc60158a70 legacy=0x7fcc600b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc600b37c0 0x7fcc4c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc4c003f30 0x7fcc4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.616+0000 7fcc57fff640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc4c004d40 0x7fcc4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.617+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6015c210 con 0x7fcc4c004d40 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.617+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcc600b2ce0 con 0x7fcc4c003f30 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.617+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcc4c00b9c0 con 0x7fcc600b37c0 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.617+0000 7fcc65e25640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc4c004d40 0x7fcc4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.617+0000 7fcc64e23640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc4c003f30 0x7fcc4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.617+0000 7fcc65624640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc600b37c0 0x7fcc4c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.617+0000 7fcc65e25640 1 -- 172.21.15.77:0/66307547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc600b37c0 msgr2=0x7fcc4c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.617+0000 7fcc65e25640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc600b37c0 0x7fcc4c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.617+0000 7fcc65e25640 1 -- 172.21.15.77:0/66307547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc4c003f30 msgr2=0x7fcc4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.617+0000 7fcc65e25640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc4c003f30 0x7fcc4c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.617+0000 7fcc65e25640 1 -- 172.21.15.77:0/66307547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4c003520 con 0x7fcc4c004d40 2024-04-25T01:22:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.618+0000 7fcc65e25640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc4c004d40 0x7fcc4c0051c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcc58009fc0 tx=0x7fcc4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65dfd27ffb6c09be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.618+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58009c10 con 0x7fcc4c004d40 2024-04-25T01:22:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.618+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc58036070 con 0x7fcc4c004d40 2024-04-25T01:22:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.618+0000 7fcc57fff640 1 -- 172.21.15.77:0/66307547 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58031670 con 0x7fcc4c004d40 2024-04-25T01:22:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.618+0000 7fcc67888640 1 -- 172.21.15.77:0/66307547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc4c004d40 msgr2=0x7fcc4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.618+0000 7fcc67888640 1 --2- 172.21.15.77:0/66307547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc4c004d40 0x7fcc4c0051c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcc58009fc0 tx=0x7fcc4c005830 comp rx=0 tx=0).stop 2024-04-25T01:22:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.618+0000 7fcc65e25640 1 -- 172.21.15.77:0/66307547 reap_dead start 2024-04-25T01:22:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.618+0000 7fcc67888640 1 -- 172.21.15.77:0/66307547 shutdown_connections 2024-04-25T01:22:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.618+0000 7fcc67888640 1 -- 172.21.15.77:0/66307547 >> 172.21.15.77:0/66307547 conn(0x7fcc6006b720 msgr2=0x7fcc600b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.618+0000 7fcc67888640 1 -- 172.21.15.77:0/66307547 shutdown_connections 2024-04-25T01:22:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.618+0000 7fcc67888640 1 -- 172.21.15.77:0/66307547 wait complete. 2024-04-25T01:22:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.619+0000 7fcc67888640 1 Processor -- start 2024-04-25T01:22:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.619+0000 7fcc67888640 1 -- start start 2024-04-25T01:22:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.620+0000 7fcc67888640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc600cc250 0x7fcc60154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.620+0000 7fcc67888640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc60156190 0x7fcc60152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.620+0000 7fcc67888640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc60150260 0x7fcc601506e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.620+0000 7fcc67888640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6015ba60 con 0x7fcc60150260 2024-04-25T01:22:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.620+0000 7fcc67888640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcc60158e50 con 0x7fcc600cc250 2024-04-25T01:22:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.620+0000 7fcc67888640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcc6015bbe0 con 0x7fcc60156190 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.620+0000 7fcc65e25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc60150260 0x7fcc601506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.620+0000 7fcc65e25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc60150260 0x7fcc601506e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48224/0 (socket says 172.21.15.77:48224) 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.620+0000 7fcc65e25640 1 -- 172.21.15.77:0/1271370722 learned_addr learned my addr 172.21.15.77:0/1271370722 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.620+0000 7fcc64e23640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc60156190 0x7fcc60152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.620+0000 7fcc65624640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc600cc250 0x7fcc60154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.621+0000 7fcc65e25640 1 -- 172.21.15.77:0/1271370722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc60156190 msgr2=0x7fcc60152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.621+0000 7fcc65e25640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc60156190 0x7fcc60152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.621+0000 7fcc65e25640 1 -- 172.21.15.77:0/1271370722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc600cc250 msgr2=0x7fcc60154b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.621+0000 7fcc65e25640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc600cc250 0x7fcc60154b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.621+0000 7fcc65e25640 1 -- 172.21.15.77:0/1271370722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc601550f0 con 0x7fcc60150260 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.621+0000 7fcc65e25640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc60150260 0x7fcc601506e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcc58033040 tx=0x7fcc58009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.621+0000 7fcc55ffb640 1 -- 172.21.15.77:0/1271370722 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58009c10 con 0x7fcc60150260 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.621+0000 7fcc55ffb640 1 -- 172.21.15.77:0/1271370722 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc58031dd0 con 0x7fcc60150260 2024-04-25T01:22:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.621+0000 7fcc67888640 1 -- 172.21.15.77:0/1271370722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc601553d0 con 0x7fcc60150260 2024-04-25T01:22:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.621+0000 7fcc55ffb640 1 -- 172.21.15.77:0/1271370722 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58003bd0 con 0x7fcc60150260 2024-04-25T01:22:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.621+0000 7fcc67888640 1 -- 172.21.15.77:0/1271370722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc60155990 con 0x7fcc60150260 2024-04-25T01:22:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.622+0000 7fcc67888640 1 -- 172.21.15.77:0/1271370722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc60068fb0 con 0x7fcc60150260 2024-04-25T01:22:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.623+0000 7fcc55ffb640 1 -- 172.21.15.77:0/1271370722 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcc58005d10 con 0x7fcc60150260 2024-04-25T01:22:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.623+0000 7fcc55ffb640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcc4406be60 0x7fcc4406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.623+0000 7fcc55ffb640 1 -- 172.21.15.77:0/1271370722 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fcc580ac010 con 0x7fcc60150260 2024-04-25T01:22:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.623+0000 7fcc65624640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcc4406be60 0x7fcc4406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.623+0000 7fcc65624640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcc4406be60 0x7fcc4406e320 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c001eb0 tx=0x7fcc5c007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.627+0000 7fcc55ffb640 1 -- 172.21.15.77:0/1271370722 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcc58049060 con 0x7fcc60150260 2024-04-25T01:22:43.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.799+0000 7fcc67888640 1 -- 172.21.15.77:0/1271370722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcc600b3c70 con 0x7fcc60150260 2024-04-25T01:22:43.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.799+0000 7fcc55ffb640 1 -- 172.21.15.77:0/1271370722 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fcc5807a670 con 0x7fcc60150260 2024-04-25T01:22:43.781 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 -- 172.21.15.77:0/1271370722 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcc4406be60 msgr2=0x7fcc4406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcc4406be60 0x7fcc4406e320 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c001eb0 tx=0x7fcc5c007040 comp rx=0 tx=0).stop 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 -- 172.21.15.77:0/1271370722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc60150260 msgr2=0x7fcc601506e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc60150260 0x7fcc601506e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcc58033040 tx=0x7fcc58009750 comp rx=0 tx=0).stop 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 -- 172.21.15.77:0/1271370722 shutdown_connections 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcc4406be60 0x7fcc4406e320 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc60150260 0x7fcc601506e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc60156190 0x7fcc60152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 --2- 172.21.15.77:0/1271370722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc600cc250 0x7fcc60154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 -- 172.21.15.77:0/1271370722 >> 172.21.15.77:0/1271370722 conn(0x7fcc6006b720 msgr2=0x7fcc60159270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 -- 172.21.15.77:0/1271370722 shutdown_connections 2024-04-25T01:22:43.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.801+0000 7fcc67888640 1 -- 172.21.15.77:0/1271370722 wait complete. 2024-04-25T01:22:43.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:22:43.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.911+0000 7f1f8f279640 1 Processor -- start 2024-04-25T01:22:43.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.911+0000 7f1f8f279640 1 -- start start 2024-04-25T01:22:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f8f279640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88158a70 0x7f1f880b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f8f279640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8815c210 con 0x7f1f880b37c0 2024-04-25T01:22:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f8f279640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f880b2ce0 con 0x7f1f8815bc30 2024-04-25T01:22:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f8f279640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f880b2fc0 con 0x7f1f88158a70 2024-04-25T01:22:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f8c814640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1f880b37c0 0x7f1f8815b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33100/0 (socket says 172.21.15.77:33100) 2024-04-25T01:22:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f8c814640 1 -- 172.21.15.77:0/1372332880 learned_addr learned my addr 172.21.15.77:0/1372332880 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f8d015640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88158a70 0x7f1f880b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3995418963 0 0) 0x7f1f8815c210 con 0x7f1f880b37c0 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f74003520 con 0x7f1f880b37c0 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1608497123 0 0) 0x7f1f74003520 con 0x7f1f880b37c0 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88158a70 msgr2=0x7f1f880b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f7ffff640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88158a70 0x7f1f880b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 >> v1:172.21.15.121:6789/0 conn(0x7f1f8815bc30 legacy=0x7f1f8806c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f880b32a0 con 0x7f1f880b37c0 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.912+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f1f78001670 con 0x7f1f880b37c0 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 >> v1:172.21.15.77:6789/0 conn(0x7f1f880b37c0 legacy=0x7f1f8815b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f7ffff640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f740039e0 0x7f1f74003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f7ffff640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 0x7f1f74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f7ffff640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88158a70 0x7f1f740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1f880b2fc0 con 0x7f1f74004800 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7400ba70 con 0x7f1f740039e0 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f7400de80 con 0x7f1f88158a70 2024-04-25T01:22:43.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f8c814640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 0x7f1f74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f8d015640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f740039e0 0x7f1f74003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f8d816640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88158a70 0x7f1f740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f8c814640 1 -- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88158a70 msgr2=0x7f1f740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f8c814640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88158a70 0x7f1f740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f8c814640 1 -- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f740039e0 msgr2=0x7f1f74003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f8c814640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f740039e0 0x7f1f74003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f8c814640 1 -- 172.21.15.77:0/1372332880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f74003520 con 0x7f1f74004800 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.913+0000 7f1f8d816640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88158a70 0x7f1f740051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.914+0000 7f1f8c814640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 0x7f1f74004c80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800c210 tx=0x7f1f74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a651356ba439e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.914+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f78007cf0 con 0x7f1f74004800 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.914+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f78005ad0 con 0x7f1f74004800 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.914+0000 7f1f7ffff640 1 -- 172.21.15.77:0/1372332880 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f78007070 con 0x7f1f74004800 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.914+0000 7f1f8f279640 1 -- 172.21.15.77:0/1372332880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 msgr2=0x7f1f74004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.914+0000 7f1f8f279640 1 --2- 172.21.15.77:0/1372332880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 0x7f1f74004c80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800c210 tx=0x7f1f74005830 comp rx=0 tx=0).stop 2024-04-25T01:22:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.914+0000 7f1f8d816640 1 -- 172.21.15.77:0/1372332880 reap_dead start 2024-04-25T01:22:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.914+0000 7f1f8f279640 1 -- 172.21.15.77:0/1372332880 shutdown_connections 2024-04-25T01:22:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.914+0000 7f1f8f279640 1 -- 172.21.15.77:0/1372332880 >> 172.21.15.77:0/1372332880 conn(0x7f1f8806b720 msgr2=0x7f1f880b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.914+0000 7f1f8f279640 1 -- 172.21.15.77:0/1372332880 shutdown_connections 2024-04-25T01:22:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.914+0000 7f1f8f279640 1 -- 172.21.15.77:0/1372332880 wait complete. 2024-04-25T01:22:43.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.915+0000 7f1f8f279640 1 Processor -- start 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.915+0000 7f1f8f279640 1 -- start start 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8f279640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 0x7f1f88157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8f279640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88141a50 0x7f1f880b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8f279640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f880b06b0 0x7f1f881f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8f279640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8806d280 con 0x7f1f74004800 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8f279640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8806d100 con 0x7f1f880b06b0 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8f279640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f8806d400 con 0x7f1f88141a50 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8d015640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 0x7f1f88157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8d015640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 0x7f1f88157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48240/0 (socket says 172.21.15.77:48240) 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8d015640 1 -- 172.21.15.77:0/323819010 learned_addr learned my addr 172.21.15.77:0/323819010 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8c814640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88141a50 0x7f1f880b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8d816640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f880b06b0 0x7f1f881f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8d015640 1 -- 172.21.15.77:0/323819010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88141a50 msgr2=0x7f1f880b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8d015640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88141a50 0x7f1f880b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8d015640 1 -- 172.21.15.77:0/323819010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f880b06b0 msgr2=0x7f1f881f3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8d015640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f880b06b0 0x7f1f881f3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8d015640 1 -- 172.21.15.77:0/323819010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f880b0ba0 con 0x7f1f74004800 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8c814640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88141a50 0x7f1f880b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.916+0000 7f1f8d816640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f880b06b0 0x7f1f881f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.917+0000 7f1f8d015640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 0x7f1f88157830 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1f840119a0 tx=0x7f1f84011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.917+0000 7f1f7dffb640 1 -- 172.21.15.77:0/323819010 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8400faf0 con 0x7f1f74004800 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.917+0000 7f1f7dffb640 1 -- 172.21.15.77:0/323819010 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f8400fc90 con 0x7f1f74004800 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.917+0000 7f1f8f279640 1 -- 172.21.15.77:0/323819010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f880b0e80 con 0x7f1f74004800 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.917+0000 7f1f7dffb640 1 -- 172.21.15.77:0/323819010 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f84014440 con 0x7f1f74004800 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.917+0000 7f1f8f279640 1 -- 172.21.15.77:0/323819010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f88157d70 con 0x7f1f74004800 2024-04-25T01:22:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.918+0000 7f1f8f279640 1 -- 172.21.15.77:0/323819010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f88068fb0 con 0x7f1f74004800 2024-04-25T01:22:43.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.918+0000 7f1f7dffb640 1 -- 172.21.15.77:0/323819010 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1f84012040 con 0x7f1f74004800 2024-04-25T01:22:43.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.919+0000 7f1f7dffb640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6c06be30 0x7f1f6c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:43.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.919+0000 7f1f7dffb640 1 -- 172.21.15.77:0/323819010 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f1f8408b900 con 0x7f1f74004800 2024-04-25T01:22:43.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.919+0000 7f1f8c814640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6c06be30 0x7f1f6c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:43.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.919+0000 7f1f8c814640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6c06be30 0x7f1f6c06e2f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1f78032000 tx=0x7f1f78009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:43.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:43.923+0000 7f1f7dffb640 1 -- 172.21.15.77:0/323819010 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1f84019030 con 0x7f1f74004800 2024-04-25T01:22:44.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.095+0000 7f1f8f279640 1 -- 172.21.15.77:0/323819010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1f880b3c70 con 0x7f1f74004800 2024-04-25T01:22:44.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.095+0000 7f1f7dffb640 1 -- 172.21.15.77:0/323819010 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f1f84059850 con 0x7f1f74004800 2024-04-25T01:22:44.078 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:44.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 -- 172.21.15.77:0/323819010 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6c06be30 msgr2=0x7f1f6c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6c06be30 0x7f1f6c06e2f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1f78032000 tx=0x7f1f78009030 comp rx=0 tx=0).stop 2024-04-25T01:22:44.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 -- 172.21.15.77:0/323819010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 msgr2=0x7f1f88157830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 0x7f1f88157830 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1f840119a0 tx=0x7f1f84011e60 comp rx=0 tx=0).stop 2024-04-25T01:22:44.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 -- 172.21.15.77:0/323819010 shutdown_connections 2024-04-25T01:22:44.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f6c06be30 0x7f1f6c06e2f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f880b06b0 0x7f1f881f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f88141a50 0x7f1f880b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 --2- 172.21.15.77:0/323819010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f74004800 0x7f1f88157830 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 -- 172.21.15.77:0/323819010 >> 172.21.15.77:0/323819010 conn(0x7f1f8806b720 msgr2=0x7f1f881599d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:44.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 -- 172.21.15.77:0/323819010 shutdown_connections 2024-04-25T01:22:44.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.097+0000 7f1f8f279640 1 -- 172.21.15.77:0/323819010 wait complete. 2024-04-25T01:22:44.090 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:22:44.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.209+0000 7f6f9469a640 1 Processor -- start 2024-04-25T01:22:44.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.209+0000 7f6f9469a640 1 -- start start 2024-04-25T01:22:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f9469a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f8c158a70 0x7f6f8c0b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f9469a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c0b37c0 0x7f6f8c0b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f9469a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6f8c15b520 0x7f6f8c15b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f9469a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6f8c15c210 con 0x7f6f8c158a70 2024-04-25T01:22:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f9469a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6f8c0b2ce0 con 0x7f6f8c15b520 2024-04-25T01:22:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f9469a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6f8c0b2fc0 con 0x7f6f8c0b37c0 2024-04-25T01:22:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f92436640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f8c158a70 0x7f6f8c0b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f92436640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f8c158a70 0x7f6f8c0b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48242/0 (socket says 172.21.15.77:48242) 2024-04-25T01:22:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f92436640 1 -- 172.21.15.77:0/4217462779 learned_addr learned my addr 172.21.15.77:0/4217462779 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f91c35640 1 --2- 172.21.15.77:0/4217462779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c0b37c0 0x7f6f8c0b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f92436640 1 -- 172.21.15.77:0/4217462779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c0b37c0 msgr2=0x7f6f8c0b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f92c37640 1 --2- 172.21.15.77:0/4217462779 >> v2:172.21.15.121:3300/0 conn(0x7f6f8c15b520 0x7f6f8c15b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f92436640 1 --2- 172.21.15.77:0/4217462779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c0b37c0 0x7f6f8c0b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f92436640 1 -- 172.21.15.77:0/4217462779 >> v2:172.21.15.121:3300/0 conn(0x7f6f8c15b520 msgr2=0x7f6f8c15b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f92436640 1 --2- 172.21.15.77:0/4217462779 >> v2:172.21.15.121:3300/0 conn(0x7f6f8c15b520 0x7f6f8c15b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.210+0000 7f6f92436640 1 -- 172.21.15.77:0/4217462779 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8c0b32a0 con 0x7f6f8c158a70 2024-04-25T01:22:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.211+0000 7f6f92436640 1 --2- 172.21.15.77:0/4217462779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c158a70 0x7f6f8c0b40c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c06e990 tx=0x7f6f8802c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e5ba9420e402bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.211+0000 7f6f91434640 1 -- 172.21.15.77:0/4217462779 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88032790 con 0x7f6f8c158a70 2024-04-25T01:22:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.211+0000 7f6f91434640 1 -- 172.21.15.77:0/4217462779 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f88038070 con 0x7f6f8c158a70 2024-04-25T01:22:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.211+0000 7f6f91434640 1 -- 172.21.15.77:0/4217462779 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88032b50 con 0x7f6f8c158a70 2024-04-25T01:22:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.212+0000 7f6f9469a640 1 -- 172.21.15.77:0/4217462779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c158a70 msgr2=0x7f6f8c0b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.212+0000 7f6f9469a640 1 --2- 172.21.15.77:0/4217462779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c158a70 0x7f6f8c0b40c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c06e990 tx=0x7f6f8802c890 comp rx=0 tx=0).stop 2024-04-25T01:22:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.212+0000 7f6f9469a640 1 -- 172.21.15.77:0/4217462779 shutdown_connections 2024-04-25T01:22:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.212+0000 7f6f9469a640 1 --2- 172.21.15.77:0/4217462779 >> v2:172.21.15.121:3300/0 conn(0x7f6f8c15b520 0x7f6f8c15b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.212+0000 7f6f9469a640 1 --2- 172.21.15.77:0/4217462779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c0b37c0 0x7f6f8c0b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.212+0000 7f6f9469a640 1 --2- 172.21.15.77:0/4217462779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c158a70 0x7f6f8c0b40c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.212+0000 7f6f9469a640 1 -- 172.21.15.77:0/4217462779 >> 172.21.15.77:0/4217462779 conn(0x7f6f8c06b720 msgr2=0x7f6f8c0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.212+0000 7f6f9469a640 1 -- 172.21.15.77:0/4217462779 shutdown_connections 2024-04-25T01:22:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.212+0000 7f6f9469a640 1 -- 172.21.15.77:0/4217462779 wait complete. 2024-04-25T01:22:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.213+0000 7f6f9469a640 1 Processor -- start 2024-04-25T01:22:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.213+0000 7f6f9469a640 1 -- start start 2024-04-25T01:22:44.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f9469a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0b37c0 0x7f6f8c157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f9469a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c158a70 0x7f6f8c0af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f9469a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c15b520 0x7f6f8c0b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f9469a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c15eb10 con 0x7f6f8c15b520 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f9469a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c15bef0 con 0x7f6f8c0b37c0 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f9469a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6f8c15ec90 con 0x7f6f8c158a70 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f92c37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c15b520 0x7f6f8c0b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f92c37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c15b520 0x7f6f8c0b0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48258/0 (socket says 172.21.15.77:48258) 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f92c37640 1 -- 172.21.15.77:0/1942595141 learned_addr learned my addr 172.21.15.77:0/1942595141 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f92436640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0b37c0 0x7f6f8c157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f91c35640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c158a70 0x7f6f8c0af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f92c37640 1 -- 172.21.15.77:0/1942595141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c158a70 msgr2=0x7f6f8c0af7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f92c37640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c158a70 0x7f6f8c0af7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f92c37640 1 -- 172.21.15.77:0/1942595141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0b37c0 msgr2=0x7f6f8c157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f92c37640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0b37c0 0x7f6f8c157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f92c37640 1 -- 172.21.15.77:0/1942595141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8c157d70 con 0x7f6f8c15b520 2024-04-25T01:22:44.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f92c37640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c15b520 0x7f6f8c0b0250 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6f8400cc60 tx=0x7f6f8400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f82ffd640 1 -- 172.21.15.77:0/1942595141 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f84003990 con 0x7f6f8c15b520 2024-04-25T01:22:44.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f82ffd640 1 -- 172.21.15.77:0/1942595141 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f84003b30 con 0x7f6f8c15b520 2024-04-25T01:22:44.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.214+0000 7f6f9469a640 1 -- 172.21.15.77:0/1942595141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8c1331e0 con 0x7f6f8c15b520 2024-04-25T01:22:44.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.215+0000 7f6f82ffd640 1 -- 172.21.15.77:0/1942595141 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f84004020 con 0x7f6f8c15b520 2024-04-25T01:22:44.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.215+0000 7f6f9469a640 1 -- 172.21.15.77:0/1942595141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8c1337a0 con 0x7f6f8c15b520 2024-04-25T01:22:44.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.216+0000 7f6f9469a640 1 -- 172.21.15.77:0/1942595141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f8c068fb0 con 0x7f6f8c15b520 2024-04-25T01:22:44.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.216+0000 7f6f82ffd640 1 -- 172.21.15.77:0/1942595141 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6f840041c0 con 0x7f6f8c15b520 2024-04-25T01:22:44.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.216+0000 7f6f82ffd640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f7006be30 0x7f6f7006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.217+0000 7f6f82ffd640 1 -- 172.21.15.77:0/1942595141 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f6f8408b7a0 con 0x7f6f8c15b520 2024-04-25T01:22:44.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.217+0000 7f6f92436640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f7006be30 0x7f6f7006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.217+0000 7f6f92436640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f7006be30 0x7f6f7006e2f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6f88037000 tx=0x7f6f8803e040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.221+0000 7f6f82ffd640 1 -- 172.21.15.77:0/1942595141 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6f8c157d70 con 0x7f6f8c15b520 2024-04-25T01:22:44.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.392+0000 7f6f9469a640 1 -- 172.21.15.77:0/1942595141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6f8c1448d0 con 0x7f6f8c15b520 2024-04-25T01:22:44.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.393+0000 7f6f82ffd640 1 -- 172.21.15.77:0/1942595141 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f6f84014030 con 0x7f6f8c15b520 2024-04-25T01:22:44.374 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:22:44.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 -- 172.21.15.77:0/1942595141 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f7006be30 msgr2=0x7f6f7006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f7006be30 0x7f6f7006e2f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6f88037000 tx=0x7f6f8803e040 comp rx=0 tx=0).stop 2024-04-25T01:22:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 -- 172.21.15.77:0/1942595141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c15b520 msgr2=0x7f6f8c0b0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c15b520 0x7f6f8c0b0250 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6f8400cc60 tx=0x7f6f8400a5c0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 -- 172.21.15.77:0/1942595141 shutdown_connections 2024-04-25T01:22:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6f7006be30 0x7f6f7006e2f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c15b520 0x7f6f8c0b0250 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c158a70 0x7f6f8c0af7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 --2- 172.21.15.77:0/1942595141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0b37c0 0x7f6f8c157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 -- 172.21.15.77:0/1942595141 >> 172.21.15.77:0/1942595141 conn(0x7f6f8c06b720 msgr2=0x7f6f8c0b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 -- 172.21.15.77:0/1942595141 shutdown_connections 2024-04-25T01:22:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.395+0000 7f6f9469a640 1 -- 172.21.15.77:0/1942595141 wait complete. 2024-04-25T01:22:44.387 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:22:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.504+0000 7fd1dc01b640 1 Processor -- start 2024-04-25T01:22:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.505+0000 7fd1dc01b640 1 -- start start 2024-04-25T01:22:44.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.505+0000 7fd1dc01b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd1d406da80 0x7fd1d40b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.505+0000 7fd1dc01b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d415bb80 con 0x7fd1d406caf0 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.505+0000 7fd1dc01b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d40724a0 con 0x7fd1d40720c0 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.505+0000 7fd1dc01b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd1d40b3750 con 0x7fd1d406da80 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d95b6640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd1d406caf0 0x7fd1d4070b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33110/0 (socket says 172.21.15.77:33110) 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d95b6640 1 -- 172.21.15.77:0/2624024745 learned_addr learned my addr 172.21.15.77:0/2624024745 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4159546496 0 0) 0x7fd1d415bb80 con 0x7fd1d406caf0 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1c0003520 con 0x7fd1d406caf0 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1da5b8640 1 --2- 172.21.15.77:0/2624024745 >> v2:172.21.15.121:3300/0 conn(0x7fd1d406da80 0x7fd1d40b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3504988775 0 0) 0x7fd1c0003520 con 0x7fd1d406caf0 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 >> v2:172.21.15.121:3300/0 conn(0x7fd1d406da80 msgr2=0x7fd1d40b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 --2- 172.21.15.77:0/2624024745 >> v2:172.21.15.121:3300/0 conn(0x7fd1d406da80 0x7fd1d40b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 >> v1:172.21.15.121:6789/0 conn(0x7fd1d40720c0 legacy=0x7fd1d406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d40b3a30 con 0x7fd1d406caf0 2024-04-25T01:22:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fd1d0001670 con 0x7fd1d406caf0 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 >> v1:172.21.15.77:6789/0 conn(0x7fd1d406caf0 legacy=0x7fd1d4070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 --2- 172.21.15.77:0/2624024745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1c00039b0 0x7fd1c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 --2- 172.21.15.77:0/2624024745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 0x7fd1c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 --2- 172.21.15.77:0/2624024745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1c00051c0 0x7fd1c000cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d40b3750 con 0x7fd1c0004800 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c000d270 con 0x7fd1c00039b0 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c0005a10 con 0x7fd1c00051c0 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d95b6640 1 --2- 172.21.15.77:0/2624024745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 0x7fd1c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1d9db7640 1 --2- 172.21.15.77:0/2624024745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1c00039b0 0x7fd1c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.506+0000 7fd1da5b8640 1 --2- 172.21.15.77:0/2624024745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1c00051c0 0x7fd1c000cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.507+0000 7fd1d95b6640 1 -- 172.21.15.77:0/2624024745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1c00051c0 msgr2=0x7fd1c000cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.507+0000 7fd1d95b6640 1 --2- 172.21.15.77:0/2624024745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1c00051c0 0x7fd1c000cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.507+0000 7fd1d95b6640 1 -- 172.21.15.77:0/2624024745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1c00039b0 msgr2=0x7fd1c0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.507+0000 7fd1d95b6640 1 --2- 172.21.15.77:0/2624024745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1c00039b0 0x7fd1c0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.507+0000 7fd1da5b8640 1 -- 172.21.15.77:0/2624024745 reap_dead start 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.507+0000 7fd1d95b6640 1 -- 172.21.15.77:0/2624024745 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c0003520 con 0x7fd1c0004800 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.507+0000 7fd1d95b6640 1 --2- 172.21.15.77:0/2624024745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 0x7fd1c0004c80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000c210 tx=0x7fd1c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb17a4338ddf94ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.507+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0007d10 con 0x7fd1c0004800 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.507+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d0005ad0 con 0x7fd1c0004800 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.507+0000 7fd1d8db5640 1 -- 172.21.15.77:0/2624024745 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0007070 con 0x7fd1c0004800 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.508+0000 7fd1dc01b640 1 -- 172.21.15.77:0/2624024745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 msgr2=0x7fd1c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.508+0000 7fd1dc01b640 1 --2- 172.21.15.77:0/2624024745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 0x7fd1c0004c80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000c210 tx=0x7fd1c0005830 comp rx=0 tx=0).stop 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.508+0000 7fd1dc01b640 1 -- 172.21.15.77:0/2624024745 shutdown_connections 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.508+0000 7fd1dc01b640 1 --2- 172.21.15.77:0/2624024745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 0x7fd1c0004c80 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.508+0000 7fd1dc01b640 1 -- 172.21.15.77:0/2624024745 >> 172.21.15.77:0/2624024745 conn(0x7fd1d406b720 msgr2=0x7fd1d40b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.508+0000 7fd1dc01b640 1 -- 172.21.15.77:0/2624024745 shutdown_connections 2024-04-25T01:22:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.508+0000 7fd1dc01b640 1 -- 172.21.15.77:0/2624024745 wait complete. 2024-04-25T01:22:44.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.509+0000 7fd1dc01b640 1 Processor -- start 2024-04-25T01:22:44.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.509+0000 7fd1dc01b640 1 -- start start 2024-04-25T01:22:44.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1dc01b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 0x7fd1d413e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1dc01b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1d415ce30 0x7fd1d40cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1dc01b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1d4149fb0 0x7fd1d414a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1dc01b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d415b6a0 con 0x7fd1c0004800 2024-04-25T01:22:44.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1dc01b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d40b3ff0 con 0x7fd1d415ce30 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1dc01b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d415b820 con 0x7fd1d4149fb0 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1d9db7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 0x7fd1d413e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1d9db7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 0x7fd1d413e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48290/0 (socket says 172.21.15.77:48290) 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1d9db7640 1 -- 172.21.15.77:0/676436725 learned_addr learned my addr 172.21.15.77:0/676436725 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1d9db7640 1 -- 172.21.15.77:0/676436725 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1d4149fb0 msgr2=0x7fd1d414a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1da5b8640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1d4149fb0 0x7fd1d414a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1d9db7640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1d4149fb0 0x7fd1d414a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1d95b6640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1d415ce30 0x7fd1d40cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1d9db7640 1 -- 172.21.15.77:0/676436725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1d415ce30 msgr2=0x7fd1d40cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1d9db7640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1d415ce30 0x7fd1d40cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1d9db7640 1 -- 172.21.15.77:0/676436725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d415c6f0 con 0x7fd1c0004800 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1d95b6640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1d415ce30 0x7fd1d40cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.510+0000 7fd1d9db7640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 0x7fd1d413e090 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8007810 tx=0x7fd1c800fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.511+0000 7fd1c67fc640 1 -- 172.21.15.77:0/676436725 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80050d0 con 0x7fd1c0004800 2024-04-25T01:22:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.511+0000 7fd1c67fc640 1 -- 172.21.15.77:0/676436725 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c8018070 con 0x7fd1c0004800 2024-04-25T01:22:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.511+0000 7fd1dc01b640 1 -- 172.21.15.77:0/676436725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d415c9d0 con 0x7fd1c0004800 2024-04-25T01:22:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.511+0000 7fd1c67fc640 1 -- 172.21.15.77:0/676436725 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8005490 con 0x7fd1c0004800 2024-04-25T01:22:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.511+0000 7fd1dc01b640 1 -- 172.21.15.77:0/676436725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d414a970 con 0x7fd1c0004800 2024-04-25T01:22:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.512+0000 7fd1dc01b640 1 -- 172.21.15.77:0/676436725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1d415d210 con 0x7fd1c0004800 2024-04-25T01:22:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.512+0000 7fd1c67fc640 1 -- 172.21.15.77:0/676436725 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd1c800d730 con 0x7fd1c0004800 2024-04-25T01:22:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.512+0000 7fd1c67fc640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd1b806be60 0x7fd1b806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.512+0000 7fd1c67fc640 1 -- 172.21.15.77:0/676436725 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fd1c808af70 con 0x7fd1c0004800 2024-04-25T01:22:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.512+0000 7fd1d95b6640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd1b806be60 0x7fd1b806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.513+0000 7fd1d95b6640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd1b806be60 0x7fd1b806e320 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0032000 tx=0x7fd1d0034040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.517+0000 7fd1c67fc640 1 -- 172.21.15.77:0/676436725 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd1c80595a0 con 0x7fd1c0004800 2024-04-25T01:22:44.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.689+0000 7fd1dc01b640 1 -- 172.21.15.77:0/676436725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd1d4068fb0 con 0x7fd1c0004800 2024-04-25T01:22:44.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.690+0000 7fd1c67fc640 1 -- 172.21.15.77:0/676436725 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fd1c8058ea0 con 0x7fd1c0004800 2024-04-25T01:22:44.671 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:22:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.691+0000 7fd1dc01b640 1 -- 172.21.15.77:0/676436725 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd1b806be60 msgr2=0x7fd1b806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.691+0000 7fd1dc01b640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd1b806be60 0x7fd1b806e320 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0032000 tx=0x7fd1d0034040 comp rx=0 tx=0).stop 2024-04-25T01:22:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.691+0000 7fd1dc01b640 1 -- 172.21.15.77:0/676436725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 msgr2=0x7fd1d413e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.691+0000 7fd1dc01b640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 0x7fd1d413e090 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8007810 tx=0x7fd1c800fc00 comp rx=0 tx=0).stop 2024-04-25T01:22:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.691+0000 7fd1dc01b640 1 -- 172.21.15.77:0/676436725 shutdown_connections 2024-04-25T01:22:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.691+0000 7fd1dc01b640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd1b806be60 0x7fd1b806e320 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.691+0000 7fd1dc01b640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1d4149fb0 0x7fd1d414a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.691+0000 7fd1dc01b640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1d415ce30 0x7fd1d40cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.691+0000 7fd1dc01b640 1 --2- 172.21.15.77:0/676436725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0004800 0x7fd1d413e090 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.691+0000 7fd1dc01b640 1 -- 172.21.15.77:0/676436725 >> 172.21.15.77:0/676436725 conn(0x7fd1d406b720 msgr2=0x7fd1d40af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.692+0000 7fd1dc01b640 1 -- 172.21.15.77:0/676436725 shutdown_connections 2024-04-25T01:22:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.692+0000 7fd1dc01b640 1 -- 172.21.15.77:0/676436725 wait complete. 2024-04-25T01:22:44.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:44.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.799+0000 7fce980bd640 1 Processor -- start 2024-04-25T01:22:44.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.800+0000 7fce980bd640 1 -- start start 2024-04-25T01:22:44.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.800+0000 7fce980bd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fce9015e4c0 0x7fce900d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.800+0000 7fce980bd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce900d2d40 0x7fce900cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.800+0000 7fce980bd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce900d1e90 con 0x7fce900d4a10 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.800+0000 7fce980bd640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fce900b54a0 con 0x7fce9015e4c0 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.800+0000 7fce980bd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fce900d2130 con 0x7fce900d2d40 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.801+0000 7fce9665a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce900d2d40 0x7fce900cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.801+0000 7fce95e59640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fce9015e4c0 0x7fce900d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.801+0000 7fce95658640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fce900d4a10 0x7fce900b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:36682/0 (socket says 172.21.15.77:36682) 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.801+0000 7fce95e59640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fce9015e4c0 0x7fce900d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:58868/0 (socket says 172.21.15.77:58868) 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.801+0000 7fce9665a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce900d2d40 0x7fce900cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:44540/0 (socket says 172.21.15.77:44540) 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.801+0000 7fce95658640 1 -- 172.21.15.77:0/332287321 learned_addr learned my addr 172.21.15.77:0/332287321 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.801+0000 7fce9665a640 1 -- 172.21.15.77:0/332287321 >> v2:172.21.15.121:3300/0 conn(0x7fce9015e4c0 msgr2=0x7fce900d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.801+0000 7fce9665a640 1 --2- 172.21.15.77:0/332287321 >> v2:172.21.15.121:3300/0 conn(0x7fce9015e4c0 0x7fce900d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.801+0000 7fce9665a640 1 -- 172.21.15.77:0/332287321 >> v1:172.21.15.121:6789/0 conn(0x7fce900d4a10 legacy=0x7fce900b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.801+0000 7fce9665a640 1 -- 172.21.15.77:0/332287321 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce900b5780 con 0x7fce900d2d40 2024-04-25T01:22:44.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.801+0000 7fce95e59640 1 --2- 172.21.15.77:0/332287321 >> v2:172.21.15.121:3300/0 conn(0x7fce9015e4c0 0x7fce900d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:44.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.802+0000 7fce9665a640 1 --2- 172.21.15.77:0/332287321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce900d2d40 0x7fce900cfaf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fce880097a0 tx=0x7fce8802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cb7c7fbd8782f24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.802+0000 7fce94e57640 1 -- 172.21.15.77:0/332287321 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce88037070 con 0x7fce900d2d40 2024-04-25T01:22:44.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.802+0000 7fce94e57640 1 -- 172.21.15.77:0/332287321 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce8802ae50 con 0x7fce900d2d40 2024-04-25T01:22:44.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.802+0000 7fce94e57640 1 -- 172.21.15.77:0/332287321 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce88032660 con 0x7fce900d2d40 2024-04-25T01:22:44.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.802+0000 7fce980bd640 1 -- 172.21.15.77:0/332287321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce900d2d40 msgr2=0x7fce900cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.802+0000 7fce980bd640 1 --2- 172.21.15.77:0/332287321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce900d2d40 0x7fce900cfaf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fce880097a0 tx=0x7fce8802a100 comp rx=0 tx=0).stop 2024-04-25T01:22:44.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.803+0000 7fce980bd640 1 -- 172.21.15.77:0/332287321 shutdown_connections 2024-04-25T01:22:44.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.803+0000 7fce980bd640 1 --2- 172.21.15.77:0/332287321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce900d2d40 0x7fce900cfaf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.803+0000 7fce980bd640 1 --2- 172.21.15.77:0/332287321 >> v2:172.21.15.121:3300/0 conn(0x7fce9015e4c0 0x7fce900d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.803+0000 7fce980bd640 1 -- 172.21.15.77:0/332287321 >> 172.21.15.77:0/332287321 conn(0x7fce900d3ef0 msgr2=0x7fce900d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:44.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.803+0000 7fce980bd640 1 -- 172.21.15.77:0/332287321 shutdown_connections 2024-04-25T01:22:44.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.803+0000 7fce980bd640 1 -- 172.21.15.77:0/332287321 wait complete. 2024-04-25T01:22:44.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.804+0000 7fce980bd640 1 Processor -- start 2024-04-25T01:22:44.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.804+0000 7fce980bd640 1 -- start start 2024-04-25T01:22:44.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce980bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce900d2d40 0x7fce900c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce980bd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce900d4a10 0x7fce900cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce980bd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce9015e4c0 0x7fce90152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce980bd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fce9015ea20 con 0x7fce900d2d40 2024-04-25T01:22:44.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce980bd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fce9015e8a0 con 0x7fce900d4a10 2024-04-25T01:22:44.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce980bd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fce900b45f0 con 0x7fce9015e4c0 2024-04-25T01:22:44.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce95e59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce900d2d40 0x7fce900c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce95e59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce900d2d40 0x7fce900c9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48300/0 (socket says 172.21.15.77:48300) 2024-04-25T01:22:44.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce95e59640 1 -- 172.21.15.77:0/1957323120 learned_addr learned my addr 172.21.15.77:0/1957323120 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:44.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce95658640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce900d4a10 0x7fce900cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce9665a640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce9015e4c0 0x7fce90152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce95e59640 1 -- 172.21.15.77:0/1957323120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce9015e4c0 msgr2=0x7fce90152320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce95e59640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce9015e4c0 0x7fce90152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce95e59640 1 -- 172.21.15.77:0/1957323120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce900d4a10 msgr2=0x7fce900cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce95e59640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce900d4a10 0x7fce900cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce95e59640 1 -- 172.21.15.77:0/1957323120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce900cd1b0 con 0x7fce900d2d40 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce95658640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce900d4a10 0x7fce900cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce95e59640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce900d2d40 0x7fce900c9e60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00efc0 tx=0x7fce8c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce867fc640 1 -- 172.21.15.77:0/1957323120 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8c018020 con 0x7fce900d2d40 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce980bd640 1 -- 172.21.15.77:0/1957323120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce900cd490 con 0x7fce900d2d40 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce867fc640 1 -- 172.21.15.77:0/1957323120 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce8c014680 con 0x7fce900d2d40 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce867fc640 1 -- 172.21.15.77:0/1957323120 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8c014a50 con 0x7fce900d2d40 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.805+0000 7fce980bd640 1 -- 172.21.15.77:0/1957323120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce900cda50 con 0x7fce900d2d40 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.806+0000 7fce980bd640 1 -- 172.21.15.77:0/1957323120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce90068fb0 con 0x7fce900d2d40 2024-04-25T01:22:44.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.807+0000 7fce867fc640 1 -- 172.21.15.77:0/1957323120 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fce8c005d10 con 0x7fce900d2d40 2024-04-25T01:22:44.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.807+0000 7fce867fc640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fce7406be30 0x7fce7406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.807+0000 7fce867fc640 1 -- 172.21.15.77:0/1957323120 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fce8c08c070 con 0x7fce900d2d40 2024-04-25T01:22:44.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.807+0000 7fce95658640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fce7406be30 0x7fce7406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.808+0000 7fce95658640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fce7406be30 0x7fce7406e2f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fce900cc6f0 tx=0x7fce80009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:44.811+0000 7fce867fc640 1 -- 172.21.15.77:0/1957323120 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fce8c022030 con 0x7fce900d2d40 2024-04-25T01:22:44.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.018+0000 7fce980bd640 1 -- 172.21.15.77:0/1957323120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce900cde30 con 0x7fce900d2d40 2024-04-25T01:22:45.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.018+0000 7fce867fc640 1 -- 172.21.15.77:0/1957323120 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7fce8c010040 con 0x7fce900d2d40 2024-04-25T01:22:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:45.000 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4267907621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4267907621},{"type":"v1","addr":"172.21.15.77:6837","nonce":4267907621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/493518847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":493518847},{"type":"v1","addr":"172.21.15.121:6833","nonce":493518847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:21:45.408154+0000","modified":"2024-04-25T01:22:06.554124+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6835/2810316639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2810316639},{"type":"v1","addr":"172.21.15.121:6835","nonce":2810316639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T01:22:45.000 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 -- 172.21.15.77:0/1957323120 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fce7406be30 msgr2=0x7fce7406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fce7406be30 0x7fce7406e2f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fce900cc6f0 tx=0x7fce80009040 comp rx=0 tx=0).stop 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 -- 172.21.15.77:0/1957323120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce900d2d40 msgr2=0x7fce900c9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce900d2d40 0x7fce900c9e60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00efc0 tx=0x7fce8c00c490 comp rx=0 tx=0).stop 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 -- 172.21.15.77:0/1957323120 shutdown_connections 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fce7406be30 0x7fce7406e2f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce9015e4c0 0x7fce90152320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce900d4a10 0x7fce900cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 --2- 172.21.15.77:0/1957323120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce900d2d40 0x7fce900c9e60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 -- 172.21.15.77:0/1957323120 >> 172.21.15.77:0/1957323120 conn(0x7fce900d3ef0 msgr2=0x7fce900d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 -- 172.21.15.77:0/1957323120 shutdown_connections 2024-04-25T01:22:45.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.020+0000 7fce980bd640 1 -- 172.21.15.77:0/1957323120 wait complete. 2024-04-25T01:22:45.013 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:45.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.126+0000 7f0dadc9e640 1 Processor -- start 2024-04-25T01:22:45.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.126+0000 7f0dadc9e640 1 -- start start 2024-04-25T01:22:45.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0dadc9e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0da8143940 0x7f0da806e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0dadc9e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da8143d20 0x7f0da806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0dadc9e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0da8070e60 con 0x7f0da815ecf0 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0dadc9e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f0da8071140 con 0x7f0da8143940 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0dadc9e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0da8071420 con 0x7f0da8143d20 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0da77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0da8143940 0x7f0da806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0da77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0da8143940 0x7f0da806e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48316/0 (socket says 172.21.15.77:48316) 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0da7fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0da815ecf0 0x7f0da815f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33112/0 (socket says 172.21.15.77:33112) 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0da77fe640 1 -- 172.21.15.77:0/3235099802 learned_addr learned my addr 172.21.15.77:0/3235099802 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0da6ffd640 1 --2- 172.21.15.77:0/3235099802 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da8143d20 0x7f0da806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0da77fe640 1 -- 172.21.15.77:0/3235099802 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da8143d20 msgr2=0x7f0da806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0da77fe640 1 --2- 172.21.15.77:0/3235099802 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da8143d20 0x7f0da806eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0da77fe640 1 -- 172.21.15.77:0/3235099802 >> v1:172.21.15.77:6789/0 conn(0x7f0da815ecf0 legacy=0x7f0da815f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0da77fe640 1 -- 172.21.15.77:0/3235099802 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da8071700 con 0x7f0da8143940 2024-04-25T01:22:45.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0da67fc640 1 -- 172.21.15.77:0/3235099802 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3775860518 0 0) 0x7f0da8070e60 con 0x7f0da815ecf0 2024-04-25T01:22:45.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.127+0000 7f0da6ffd640 1 --2- 172.21.15.77:0/3235099802 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da8143d20 0x7f0da806eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:45.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.128+0000 7f0da77fe640 1 --2- 172.21.15.77:0/3235099802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0da8143940 0x7f0da806e650 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0d98009870 tx=0x7f0d9802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c21227c5646ff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.128+0000 7f0da67fc640 1 -- 172.21.15.77:0/3235099802 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d98037070 con 0x7f0da8143940 2024-04-25T01:22:45.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.128+0000 7f0da67fc640 1 -- 172.21.15.77:0/3235099802 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d98032440 con 0x7f0da8143940 2024-04-25T01:22:45.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.128+0000 7f0da67fc640 1 -- 172.21.15.77:0/3235099802 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d98032810 con 0x7f0da8143940 2024-04-25T01:22:45.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.129+0000 7f0dadc9e640 1 -- 172.21.15.77:0/3235099802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0da8143940 msgr2=0x7f0da806e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.129+0000 7f0dadc9e640 1 --2- 172.21.15.77:0/3235099802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0da8143940 0x7f0da806e650 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0d98009870 tx=0x7f0d9802a270 comp rx=0 tx=0).stop 2024-04-25T01:22:45.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.129+0000 7f0dadc9e640 1 -- 172.21.15.77:0/3235099802 shutdown_connections 2024-04-25T01:22:45.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.129+0000 7f0dadc9e640 1 --2- 172.21.15.77:0/3235099802 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da8143d20 0x7f0da806eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.129+0000 7f0dadc9e640 1 --2- 172.21.15.77:0/3235099802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0da8143940 0x7f0da806e650 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.129+0000 7f0dadc9e640 1 -- 172.21.15.77:0/3235099802 >> 172.21.15.77:0/3235099802 conn(0x7f0da80b1820 msgr2=0x7f0da80b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.129+0000 7f0dadc9e640 1 -- 172.21.15.77:0/3235099802 shutdown_connections 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.129+0000 7f0dadc9e640 1 -- 172.21.15.77:0/3235099802 wait complete. 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.130+0000 7f0dadc9e640 1 Processor -- start 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.130+0000 7f0dadc9e640 1 -- start start 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0dadc9e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da8143940 0x7f0da80ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0dadc9e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0da8143d20 0x7f0da8154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0dadc9e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da815ecf0 0x7f0da80aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0dadc9e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0da8071fb0 con 0x7f0da8143d20 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0dadc9e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0da8071e30 con 0x7f0da815ecf0 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0dadc9e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0da8072130 con 0x7f0da8143940 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0da6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0da8143d20 0x7f0da8154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0da6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0da8143d20 0x7f0da8154650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48326/0 (socket says 172.21.15.77:48326) 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0da6ffd640 1 -- 172.21.15.77:0/2482564828 learned_addr learned my addr 172.21.15.77:0/2482564828 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0da6ffd640 1 -- 172.21.15.77:0/2482564828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da8143940 msgr2=0x7f0da80ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0da6ffd640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da8143940 0x7f0da80ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0da6ffd640 1 -- 172.21.15.77:0/2482564828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da815ecf0 msgr2=0x7f0da80aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0da6ffd640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da815ecf0 0x7f0da80aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0da6ffd640 1 -- 172.21.15.77:0/2482564828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da80af1f0 con 0x7f0da8143d20 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0da6ffd640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0da8143d20 0x7f0da8154650 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00e7e0 tx=0x7f0d9c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.131+0000 7f0d8ffff640 1 -- 172.21.15.77:0/2482564828 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c00f040 con 0x7f0da8143d20 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.132+0000 7f0d8ffff640 1 -- 172.21.15.77:0/2482564828 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d9c004030 con 0x7f0da8143d20 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.132+0000 7f0dadc9e640 1 -- 172.21.15.77:0/2482564828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da80adc10 con 0x7f0da8143d20 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.132+0000 7f0d8ffff640 1 -- 172.21.15.77:0/2482564828 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c004400 con 0x7f0da8143d20 2024-04-25T01:22:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.132+0000 7f0dadc9e640 1 -- 172.21.15.77:0/2482564828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da80ae1d0 con 0x7f0da8143d20 2024-04-25T01:22:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.133+0000 7f0dadc9e640 1 -- 172.21.15.77:0/2482564828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0da8068fb0 con 0x7f0da8143d20 2024-04-25T01:22:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.133+0000 7f0d8ffff640 1 -- 172.21.15.77:0/2482564828 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0d9c00c6d0 con 0x7f0da8143d20 2024-04-25T01:22:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.134+0000 7f0d8ffff640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0d9406be30 0x7f0d9406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.134+0000 7f0da77fe640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0d9406be30 0x7f0d9406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.134+0000 7f0d8ffff640 1 -- 172.21.15.77:0/2482564828 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f0d9c08b0e0 con 0x7f0da8143d20 2024-04-25T01:22:45.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.134+0000 7f0da77fe640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0d9406be30 0x7f0d9406e2f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0d98034000 tx=0x7f0d9802c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.138+0000 7f0d8ffff640 1 -- 172.21.15.77:0/2482564828 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0d9c017020 con 0x7f0da8143d20 2024-04-25T01:22:45.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.314+0000 7f0dadc9e640 1 -- 172.21.15.77:0/2482564828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0da806d150 con 0x7f0da8143d20 2024-04-25T01:22:45.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.315+0000 7f0d8ffff640 1 -- 172.21.15.77:0/2482564828 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18922 (secure 0 0 0) 0x7f0d9c0595f0 con 0x7f0da8143d20 2024-04-25T01:22:45.296 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:45.297 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:26.287128+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:45.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 -- 172.21.15.77:0/2482564828 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0d9406be30 msgr2=0x7f0d9406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0d9406be30 0x7f0d9406e2f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0d98034000 tx=0x7f0d9802c000 comp rx=0 tx=0).stop 2024-04-25T01:22:45.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 -- 172.21.15.77:0/2482564828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0da8143d20 msgr2=0x7f0da8154650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0da8143d20 0x7f0da8154650 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00e7e0 tx=0x7f0d9c00eca0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 -- 172.21.15.77:0/2482564828 shutdown_connections 2024-04-25T01:22:45.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0d9406be30 0x7f0d9406e2f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0da8143d20 0x7f0da8154650 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0da815ecf0 0x7f0da80aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 --2- 172.21.15.77:0/2482564828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0da8143940 0x7f0da80ceb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 -- 172.21.15.77:0/2482564828 >> 172.21.15.77:0/2482564828 conn(0x7f0da80b1820 msgr2=0x7f0da8072590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:45.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 -- 172.21.15.77:0/2482564828 shutdown_connections 2024-04-25T01:22:45.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.317+0000 7f0dadc9e640 1 -- 172.21.15.77:0/2482564828 wait complete. 2024-04-25T01:22:45.309 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T01:22:45.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:45.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.426+0000 7fed1dcd4640 1 Processor -- start 2024-04-25T01:22:45.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.426+0000 7fed1dcd4640 1 -- start start 2024-04-25T01:22:45.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.426+0000 7fed1dcd4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed18143940 0x7fed1806e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.426+0000 7fed1dcd4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed1815f400 0x7fed1806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.426+0000 7fed1dcd4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed18070e60 con 0x7fed18143d20 2024-04-25T01:22:45.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.426+0000 7fed1dcd4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fed18071140 con 0x7fed18143940 2024-04-25T01:22:45.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.426+0000 7fed1dcd4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed18071420 con 0x7fed1815f400 2024-04-25T01:22:45.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.427+0000 7fed177fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed18143940 0x7fed1806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.427+0000 7fed16ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fed18143d20 0x7fed1815ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33126/0 (socket says 172.21.15.77:33126) 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.427+0000 7fed16ffd640 1 -- 172.21.15.77:0/324733767 learned_addr learned my addr 172.21.15.77:0/324733767 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.427+0000 7fed17fff640 1 --2- 172.21.15.77:0/324733767 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed1815f400 0x7fed1806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.427+0000 7fed177fe640 1 -- 172.21.15.77:0/324733767 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed1815f400 msgr2=0x7fed1806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.427+0000 7fed177fe640 1 --2- 172.21.15.77:0/324733767 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed1815f400 0x7fed1806eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.427+0000 7fed177fe640 1 -- 172.21.15.77:0/324733767 >> v1:172.21.15.77:6789/0 conn(0x7fed18143d20 legacy=0x7fed1815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.427+0000 7fed167fc640 1 -- 172.21.15.77:0/324733767 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1322022440 0 0) 0x7fed18070e60 con 0x7fed18143d20 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.427+0000 7fed177fe640 1 -- 172.21.15.77:0/324733767 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed18071700 con 0x7fed18143940 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed177fe640 1 --2- 172.21.15.77:0/324733767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed18143940 0x7fed1806e650 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00ba70 tx=0x7fed0c02c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa2dbc6c98d2928 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed167fc640 1 -- 172.21.15.77:0/324733767 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c034620 con 0x7fed18143940 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed167fc640 1 -- 172.21.15.77:0/324733767 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0c0347c0 con 0x7fed18143940 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed167fc640 1 -- 172.21.15.77:0/324733767 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c034b90 con 0x7fed18143940 2024-04-25T01:22:45.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed1dcd4640 1 -- 172.21.15.77:0/324733767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed18143940 msgr2=0x7fed1806e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed1dcd4640 1 --2- 172.21.15.77:0/324733767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed18143940 0x7fed1806e650 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00ba70 tx=0x7fed0c02c870 comp rx=0 tx=0).stop 2024-04-25T01:22:45.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed1dcd4640 1 -- 172.21.15.77:0/324733767 shutdown_connections 2024-04-25T01:22:45.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed1dcd4640 1 --2- 172.21.15.77:0/324733767 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed1815f400 0x7fed1806eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed1dcd4640 1 --2- 172.21.15.77:0/324733767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed18143940 0x7fed1806e650 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed1dcd4640 1 -- 172.21.15.77:0/324733767 >> 172.21.15.77:0/324733767 conn(0x7fed180b1820 msgr2=0x7fed180b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:45.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed1dcd4640 1 -- 172.21.15.77:0/324733767 shutdown_connections 2024-04-25T01:22:45.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.428+0000 7fed1dcd4640 1 -- 172.21.15.77:0/324733767 wait complete. 2024-04-25T01:22:45.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.429+0000 7fed1dcd4640 1 Processor -- start 2024-04-25T01:22:45.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.430+0000 7fed1dcd4640 1 -- start start 2024-04-25T01:22:45.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.430+0000 7fed1dcd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed18143940 0x7fed18156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.430+0000 7fed1dcd4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed18143d20 0x7fed18152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.430+0000 7fed1dcd4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed1815f400 0x7fed18155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.430+0000 7fed1dcd4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fed1806d8d0 con 0x7fed18143940 2024-04-25T01:22:45.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.430+0000 7fed1dcd4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fed1806d750 con 0x7fed1815f400 2024-04-25T01:22:45.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.430+0000 7fed1dcd4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed1806da50 con 0x7fed18143d20 2024-04-25T01:22:45.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.430+0000 7fed177fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed18143940 0x7fed18156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.430+0000 7fed177fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed18143940 0x7fed18156120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48348/0 (socket says 172.21.15.77:48348) 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.430+0000 7fed177fe640 1 -- 172.21.15.77:0/135650830 learned_addr learned my addr 172.21.15.77:0/135650830 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fed177fe640 1 -- 172.21.15.77:0/135650830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed18143d20 msgr2=0x7fed18152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fed177fe640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed18143d20 0x7fed18152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fed17fff640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed1815f400 0x7fed18155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fed177fe640 1 -- 172.21.15.77:0/135650830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed1815f400 msgr2=0x7fed18155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fed177fe640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed1815f400 0x7fed18155aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fed177fe640 1 -- 172.21.15.77:0/135650830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed18158930 con 0x7fed18143940 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fed177fe640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed18143940 0x7fed18156120 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00bb60 tx=0x7fed0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fecfffff640 1 -- 172.21.15.77:0/135650830 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c043070 con 0x7fed18143940 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fecfffff640 1 -- 172.21.15.77:0/135650830 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0c033c50 con 0x7fed18143940 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fed1dcd4640 1 -- 172.21.15.77:0/135650830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed18158c10 con 0x7fed18143940 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fecfffff640 1 -- 172.21.15.77:0/135650830 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c006e10 con 0x7fed18143940 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.431+0000 7fed1dcd4640 1 -- 172.21.15.77:0/135650830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed181591d0 con 0x7fed18143940 2024-04-25T01:22:45.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.432+0000 7fed1dcd4640 1 -- 172.21.15.77:0/135650830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed180af750 con 0x7fed18143940 2024-04-25T01:22:45.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.433+0000 7fecfffff640 1 -- 172.21.15.77:0/135650830 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fed0c03c040 con 0x7fed18143940 2024-04-25T01:22:45.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.433+0000 7fecfffff640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed0406be30 0x7fed0406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.433+0000 7fecfffff640 1 -- 172.21.15.77:0/135650830 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fed0c0aeac0 con 0x7fed18143940 2024-04-25T01:22:45.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.433+0000 7fed16ffd640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed0406be30 0x7fed0406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.433+0000 7fed16ffd640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed0406be30 0x7fed0406e2f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fed08007fd0 tx=0x7fed08007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.437+0000 7fecfffff640 1 -- 172.21.15.77:0/135650830 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fed180af750 con 0x7fed18143940 2024-04-25T01:22:45.625 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:22:45.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:22:45.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.648+0000 7fed1dcd4640 1 -- 172.21.15.77:0/135650830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed180af9a0 con 0x7fed18143940 2024-04-25T01:22:45.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.649+0000 7fecfffff640 1 -- 172.21.15.77:0/135650830 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7fed180af9a0 con 0x7fed18143940 2024-04-25T01:22:45.630 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:45.631 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4267907621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4267907621},{"type":"v1","addr":"172.21.15.77:6837","nonce":4267907621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/493518847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":493518847},{"type":"v1","addr":"172.21.15.121:6833","nonce":493518847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:21:45.408154+0000","modified":"2024-04-25T01:22:06.554124+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6835/2810316639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2810316639},{"type":"v1","addr":"172.21.15.121:6835","nonce":2810316639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T01:22:45.631 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2024-04-25T01:22:45.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 -- 172.21.15.77:0/135650830 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed0406be30 msgr2=0x7fed0406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed0406be30 0x7fed0406e2f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fed08007fd0 tx=0x7fed08007920 comp rx=0 tx=0).stop 2024-04-25T01:22:45.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 -- 172.21.15.77:0/135650830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed18143940 msgr2=0x7fed18156120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed18143940 0x7fed18156120 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00bb60 tx=0x7fed0c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 -- 172.21.15.77:0/135650830 shutdown_connections 2024-04-25T01:22:45.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fed0406be30 0x7fed0406e2f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed1815f400 0x7fed18155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed18143d20 0x7fed18152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 --2- 172.21.15.77:0/135650830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed18143940 0x7fed18156120 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 -- 172.21.15.77:0/135650830 >> 172.21.15.77:0/135650830 conn(0x7fed180b1820 msgr2=0x7fed1815baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:45.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 -- 172.21.15.77:0/135650830 shutdown_connections 2024-04-25T01:22:45.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.651+0000 7fed1dcd4640 1 -- 172.21.15.77:0/135650830 wait complete. 2024-04-25T01:22:45.644 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:45.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.800+0000 7f1cad71b640 1 Processor -- start 2024-04-25T01:22:45.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.800+0000 7f1cad71b640 1 -- start start 2024-04-25T01:22:45.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1cad71b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1ca80d24a0 0x7f1ca80b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1cad71b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca80d2880 0x7f1ca80d08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1cad71b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ca80b2da0 0x7f1ca8161390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1cad71b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca80d1120 con 0x7f1ca80b2da0 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1cad71b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca80d1400 con 0x7f1ca80d24a0 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1cad71b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca80d16e0 con 0x7f1ca80d2880 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1ca77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ca80b2da0 0x7f1ca8161390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1ca77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ca80b2da0 0x7f1ca8161390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48364/0 (socket says 172.21.15.77:48364) 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1ca77fe640 1 -- 172.21.15.77:0/3767172147 learned_addr learned my addr 172.21.15.77:0/3767172147 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1ca6ffd640 1 --2- 172.21.15.77:0/3767172147 >> v2:172.21.15.121:3300/0 conn(0x7f1ca80d24a0 0x7f1ca80b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1ca67fc640 1 --2- 172.21.15.77:0/3767172147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca80d2880 0x7f1ca80d08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1ca77fe640 1 -- 172.21.15.77:0/3767172147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca80d2880 msgr2=0x7f1ca80d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1ca77fe640 1 --2- 172.21.15.77:0/3767172147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca80d2880 0x7f1ca80d08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1ca77fe640 1 -- 172.21.15.77:0/3767172147 >> v2:172.21.15.121:3300/0 conn(0x7f1ca80d24a0 msgr2=0x7f1ca80b0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1ca77fe640 1 --2- 172.21.15.77:0/3767172147 >> v2:172.21.15.121:3300/0 conn(0x7f1ca80d24a0 0x7f1ca80b0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1ca77fe640 1 -- 172.21.15.77:0/3767172147 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca80cee60 con 0x7f1ca80b2da0 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.801+0000 7f1ca67fc640 1 --2- 172.21.15.77:0/3767172147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca80d2880 0x7f1ca80d08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1ca77fe640 1 --2- 172.21.15.77:0/3767172147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80b2da0 0x7f1ca8161390 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80b6af0 tx=0x7f1ca002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7540694fdaa6c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1ca5ffb640 1 -- 172.21.15.77:0/3767172147 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0036070 con 0x7f1ca80b2da0 2024-04-25T01:22:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1ca5ffb640 1 -- 172.21.15.77:0/3767172147 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca0031600 con 0x7f1ca80b2da0 2024-04-25T01:22:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1ca5ffb640 1 -- 172.21.15.77:0/3767172147 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca00319d0 con 0x7f1ca80b2da0 2024-04-25T01:22:45.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1cad71b640 1 -- 172.21.15.77:0/3767172147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80b2da0 msgr2=0x7f1ca8161390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1cad71b640 1 --2- 172.21.15.77:0/3767172147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80b2da0 0x7f1ca8161390 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80b6af0 tx=0x7f1ca002a6a0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1cad71b640 1 -- 172.21.15.77:0/3767172147 shutdown_connections 2024-04-25T01:22:45.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1cad71b640 1 --2- 172.21.15.77:0/3767172147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80b2da0 0x7f1ca8161390 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1cad71b640 1 --2- 172.21.15.77:0/3767172147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca80d2880 0x7f1ca80d08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1cad71b640 1 --2- 172.21.15.77:0/3767172147 >> v2:172.21.15.121:3300/0 conn(0x7f1ca80d24a0 0x7f1ca80b0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1cad71b640 1 -- 172.21.15.77:0/3767172147 >> 172.21.15.77:0/3767172147 conn(0x7f1ca809d200 msgr2=0x7f1ca80d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.802+0000 7f1cad71b640 1 -- 172.21.15.77:0/3767172147 shutdown_connections 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.803+0000 7f1cad71b640 1 -- 172.21.15.77:0/3767172147 wait complete. 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.804+0000 7f1cad71b640 1 Processor -- start 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.804+0000 7f1cad71b640 1 -- start start 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.804+0000 7f1cad71b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80b2da0 0x7f1ca80ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.804+0000 7f1cad71b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca80d24a0 0x7f1ca80b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.804+0000 7f1cad71b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ca80d2880 0x7f1ca815e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.804+0000 7f1cad71b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca81618d0 con 0x7f1ca80b2da0 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.804+0000 7f1cad71b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca80ceb10 con 0x7f1ca80d2880 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.804+0000 7f1cad71b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca8161a50 con 0x7f1ca80d24a0 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80b2da0 0x7f1ca80ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80b2da0 0x7f1ca80ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48380/0 (socket says 172.21.15.77:48380) 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca6ffd640 1 -- 172.21.15.77:0/1211202778 learned_addr learned my addr 172.21.15.77:0/1211202778 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca77fe640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ca80d2880 0x7f1ca815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca67fc640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca80d24a0 0x7f1ca80b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca6ffd640 1 -- 172.21.15.77:0/1211202778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca80d24a0 msgr2=0x7f1ca80b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca6ffd640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca80d24a0 0x7f1ca80b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca6ffd640 1 -- 172.21.15.77:0/1211202778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ca80d2880 msgr2=0x7f1ca815e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca6ffd640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ca80d2880 0x7f1ca815e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca6ffd640 1 -- 172.21.15.77:0/1211202778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca80cd470 con 0x7f1ca80b2da0 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca77fe640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ca80d2880 0x7f1ca815e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1ca6ffd640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80b2da0 0x7f1ca80ccde0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800d800 tx=0x7f1c9800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1c8f7fe640 1 -- 172.21.15.77:0/1211202778 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9800b6c0 con 0x7f1ca80b2da0 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1c8f7fe640 1 -- 172.21.15.77:0/1211202778 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c98019070 con 0x7f1ca80b2da0 2024-04-25T01:22:45.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1c8f7fe640 1 -- 172.21.15.77:0/1211202778 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9800ba80 con 0x7f1ca80b2da0 2024-04-25T01:22:45.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1cad71b640 1 -- 172.21.15.77:0/1211202778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca80c7ad0 con 0x7f1ca80b2da0 2024-04-25T01:22:45.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.805+0000 7f1cad71b640 1 -- 172.21.15.77:0/1211202778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca80c8370 con 0x7f1ca80b2da0 2024-04-25T01:22:45.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.806+0000 7f1cad71b640 1 -- 172.21.15.77:0/1211202778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca80c7ca0 con 0x7f1ca80b2da0 2024-04-25T01:22:45.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.807+0000 7f1c8f7fe640 1 -- 172.21.15.77:0/1211202778 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1c9800bc20 con 0x7f1ca80b2da0 2024-04-25T01:22:45.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.807+0000 7f1c8f7fe640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1c9406be30 0x7f1c9406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.807+0000 7f1c8f7fe640 1 -- 172.21.15.77:0/1211202778 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f1c9808b6b0 con 0x7f1ca80b2da0 2024-04-25T01:22:45.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.807+0000 7f1ca67fc640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1c9406be30 0x7f1c9406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.807+0000 7f1ca67fc640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1c9406be30 0x7f1c9406e2f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1ca806bd30 tx=0x7f1c9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:45.811+0000 7f1c8f7fe640 1 -- 172.21.15.77:0/1211202778 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1c98059ce0 con 0x7f1ca80b2da0 2024-04-25T01:22:46.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.020+0000 7f1cad71b640 1 -- 172.21.15.77:0/1211202778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ca8068fb0 con 0x7f1ca80b2da0 2024-04-25T01:22:46.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.020+0000 7f1c8f7fe640 1 -- 172.21.15.77:0/1211202778 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7f1c980595e0 con 0x7f1ca80b2da0 2024-04-25T01:22:46.002 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:46.002 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3931712462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3931712462},{"type":"v1","addr":"172.21.15.77:6836","nonce":3931712462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4267907621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4267907621},{"type":"v1","addr":"172.21.15.77:6837","nonce":4267907621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/493518847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":493518847},{"type":"v1","addr":"172.21.15.121:6833","nonce":493518847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:21:45.408154+0000","modified":"2024-04-25T01:22:06.554124+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6835/2810316639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2810316639},{"type":"v1","addr":"172.21.15.121:6835","nonce":2810316639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T01:22:46.002 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2024-04-25T01:22:46.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.022+0000 7f1cad71b640 1 -- 172.21.15.77:0/1211202778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1c9406be30 msgr2=0x7f1c9406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.022+0000 7f1cad71b640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1c9406be30 0x7f1c9406e2f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1ca806bd30 tx=0x7f1c9c008040 comp rx=0 tx=0).stop 2024-04-25T01:22:46.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.022+0000 7f1cad71b640 1 -- 172.21.15.77:0/1211202778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80b2da0 msgr2=0x7f1ca80ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.022+0000 7f1cad71b640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80b2da0 0x7f1ca80ccde0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800d800 tx=0x7f1c9800dcc0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.023+0000 7f1cad71b640 1 -- 172.21.15.77:0/1211202778 shutdown_connections 2024-04-25T01:22:46.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.023+0000 7f1cad71b640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1c9406be30 0x7f1c9406e2f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.023+0000 7f1cad71b640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ca80d2880 0x7f1ca815e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.023+0000 7f1cad71b640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca80d24a0 0x7f1ca80b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.023+0000 7f1cad71b640 1 --2- 172.21.15.77:0/1211202778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80b2da0 0x7f1ca80ccde0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.023+0000 7f1cad71b640 1 -- 172.21.15.77:0/1211202778 >> 172.21.15.77:0/1211202778 conn(0x7f1ca809d200 msgr2=0x7f1ca806d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:46.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.023+0000 7f1cad71b640 1 -- 172.21.15.77:0/1211202778 shutdown_connections 2024-04-25T01:22:46.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.023+0000 7f1cad71b640 1 -- 172.21.15.77:0/1211202778 wait complete. 2024-04-25T01:22:46.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:46.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.132+0000 7f1f633fb640 1 Processor -- start 2024-04-25T01:22:46.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.132+0000 7f1f633fb640 1 -- start start 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f633fb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1f5c15e4c0 0x7f1f5c0d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f633fb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0d4a10 0x7f1f5c0b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f633fb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f5c0d1e90 con 0x7f1f5c0d2d40 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f633fb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f1f5c0b54a0 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f633fb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f5c0d2130 con 0x7f1f5c0d4a10 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f61197640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1f5c15e4c0 0x7f1f5c0d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f61998640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1f5c0d2d40 0x7f1f5c0d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33136/0 (socket says 172.21.15.77:33136) 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f61998640 1 -- 172.21.15.77:0/3554200539 learned_addr learned my addr 172.21.15.77:0/3554200539 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f61197640 1 -- 172.21.15.77:0/3554200539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0d4a10 msgr2=0x7f1f5c0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f61197640 1 --2- 172.21.15.77:0/3554200539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0d4a10 0x7f1f5c0b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f61197640 1 -- 172.21.15.77:0/3554200539 >> v1:172.21.15.77:6789/0 conn(0x7f1f5c0d2d40 legacy=0x7f1f5c0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.133+0000 7f1f61197640 1 -- 172.21.15.77:0/3554200539 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5c0b5780 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.134+0000 7f1f61197640 1 --2- 172.21.15.77:0/3554200539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f5c15e4c0 0x7f1f5c0d2800 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009d90 tx=0x7f1f5802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e927c73dc077d192 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.134+0000 7f1f53fff640 1 -- 172.21.15.77:0/3554200539 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58031600 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.134+0000 7f1f53fff640 1 -- 172.21.15.77:0/3554200539 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f580317a0 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.134+0000 7f1f53fff640 1 -- 172.21.15.77:0/3554200539 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58031b70 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.135+0000 7f1f633fb640 1 -- 172.21.15.77:0/3554200539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f5c15e4c0 msgr2=0x7f1f5c0d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.135+0000 7f1f633fb640 1 --2- 172.21.15.77:0/3554200539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f5c15e4c0 0x7f1f5c0d2800 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009d90 tx=0x7f1f5802a5d0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.135+0000 7f1f633fb640 1 -- 172.21.15.77:0/3554200539 shutdown_connections 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.135+0000 7f1f633fb640 1 --2- 172.21.15.77:0/3554200539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0d4a10 0x7f1f5c0b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.135+0000 7f1f633fb640 1 --2- 172.21.15.77:0/3554200539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f5c15e4c0 0x7f1f5c0d2800 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.135+0000 7f1f633fb640 1 -- 172.21.15.77:0/3554200539 >> 172.21.15.77:0/3554200539 conn(0x7f1f5c0d3ef0 msgr2=0x7f1f5c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.135+0000 7f1f633fb640 1 -- 172.21.15.77:0/3554200539 shutdown_connections 2024-04-25T01:22:46.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.135+0000 7f1f633fb640 1 -- 172.21.15.77:0/3554200539 wait complete. 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.136+0000 7f1f633fb640 1 Processor -- start 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.136+0000 7f1f633fb640 1 -- start start 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f633fb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0d2d40 0x7f1f5c153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f633fb640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f5c0d4a10 0x7f1f5c153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f633fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f5c15e4c0 0x7f1f5c0b7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f633fb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c15ea20 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f633fb640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c15e8a0 con 0x7f1f5c0d4a10 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f633fb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f5c0d1010 con 0x7f1f5c0d2d40 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f61998640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f5c15e4c0 0x7f1f5c0b7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f61998640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f5c15e4c0 0x7f1f5c0b7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48394/0 (socket says 172.21.15.77:48394) 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f61998640 1 -- 172.21.15.77:0/847041584 learned_addr learned my addr 172.21.15.77:0/847041584 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f60996640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f5c0d4a10 0x7f1f5c153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f61197640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0d2d40 0x7f1f5c153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f61998640 1 -- 172.21.15.77:0/847041584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0d2d40 msgr2=0x7f1f5c153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f61998640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0d2d40 0x7f1f5c153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f61998640 1 -- 172.21.15.77:0/847041584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f5c0d4a10 msgr2=0x7f1f5c153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f61998640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f5c0d4a10 0x7f1f5c153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f61998640 1 -- 172.21.15.77:0/847041584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5c0cc060 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f60996640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f5c0d4a10 0x7f1f5c153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f61998640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f5c15e4c0 0x7f1f5c0b7500 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400c420 tx=0x7f1f5400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f51ffb640 1 -- 172.21.15.77:0/847041584 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f54017070 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f51ffb640 1 -- 172.21.15.77:0/847041584 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f54005ce0 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f633fb640 1 -- 172.21.15.77:0/847041584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f5c0cc340 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.137+0000 7f1f51ffb640 1 -- 172.21.15.77:0/847041584 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f540072d0 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.138+0000 7f1f633fb640 1 -- 172.21.15.77:0/847041584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f5c0cc900 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.139+0000 7f1f633fb640 1 -- 172.21.15.77:0/847041584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f5c068fb0 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.139+0000 7f1f51ffb640 1 -- 172.21.15.77:0/847041584 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1f5401a070 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.139+0000 7f1f51ffb640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f4006be30 0x7f1f4006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.139+0000 7f1f51ffb640 1 -- 172.21.15.77:0/847041584 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7f1f5408a5e0 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.139+0000 7f1f61197640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f4006be30 0x7f1f4006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.140+0000 7f1f61197640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f4006be30 0x7f1f4006e2f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1f58034000 tx=0x7f1f5802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.144+0000 7f1f51ffb640 1 -- 172.21.15.77:0/847041584 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1f540584a0 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.320+0000 7f1f633fb640 1 -- 172.21.15.77:0/847041584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f5c0ce200 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.321+0000 7f1f51ffb640 1 -- 172.21.15.77:0/847041584 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18922 (secure 0 0 0) 0x7f1f5405c7a0 con 0x7f1f5c15e4c0 2024-04-25T01:22:46.303 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:46.303 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:26.287128+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:21:43.207433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:21:44.385674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:21:48.116140+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1},{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:46.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.323+0000 7f1f633fb640 1 -- 172.21.15.77:0/847041584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f4006be30 msgr2=0x7f1f4006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.323+0000 7f1f633fb640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f4006be30 0x7f1f4006e2f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1f58034000 tx=0x7f1f5802f000 comp rx=0 tx=0).stop 2024-04-25T01:22:46.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.323+0000 7f1f633fb640 1 -- 172.21.15.77:0/847041584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f5c15e4c0 msgr2=0x7f1f5c0b7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.323+0000 7f1f633fb640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f5c15e4c0 0x7f1f5c0b7500 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400c420 tx=0x7f1f5400c8e0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.324+0000 7f1f633fb640 1 -- 172.21.15.77:0/847041584 shutdown_connections 2024-04-25T01:22:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.324+0000 7f1f633fb640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1f4006be30 0x7f1f4006e2f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.324+0000 7f1f633fb640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f5c15e4c0 0x7f1f5c0b7500 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.324+0000 7f1f633fb640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f5c0d4a10 0x7f1f5c153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.324+0000 7f1f633fb640 1 --2- 172.21.15.77:0/847041584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0d2d40 0x7f1f5c153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.324+0000 7f1f633fb640 1 -- 172.21.15.77:0/847041584 >> 172.21.15.77:0/847041584 conn(0x7f1f5c0d3ef0 msgr2=0x7f1f5c0d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.324+0000 7f1f633fb640 1 -- 172.21.15.77:0/847041584 shutdown_connections 2024-04-25T01:22:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.324+0000 7f1f633fb640 1 -- 172.21.15.77:0/847041584 wait complete. 2024-04-25T01:22:46.316 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T01:22:46.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.435+0000 7fb1a2516640 1 Processor -- start 2024-04-25T01:22:46.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.435+0000 7fb1a2516640 1 -- start start 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb1a2516640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb19c15fe10 0x7fb19c06d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb1a2516640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb19c0b4240 con 0x7fb19c15f9b0 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb1a2516640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb19c0b39a0 con 0x7fb19c145590 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb1a2516640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb19c160830 con 0x7fb19c15fe10 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb19c15f9b0 0x7fb19c06d270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33150/0 (socket says 172.21.15.77:33150) 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19b7fe640 1 -- 172.21.15.77:0/2559664925 learned_addr learned my addr 172.21.15.77:0/2559664925 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217099712 0 0) 0x7fb19c0b4240 con 0x7fb19c15f9b0 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb1a0ab3640 1 --2- 172.21.15.77:0/2559664925 >> v2:172.21.15.121:3300/0 conn(0x7fb19c15fe10 0x7fb19c06d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb184003520 con 0x7fb19c15f9b0 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3144646295 0 0) 0x7fb184003520 con 0x7fb19c15f9b0 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 >> v2:172.21.15.121:3300/0 conn(0x7fb19c15fe10 msgr2=0x7fb19c06d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 --2- 172.21.15.77:0/2559664925 >> v2:172.21.15.121:3300/0 conn(0x7fb19c15fe10 0x7fb19c06d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 >> v1:172.21.15.121:6789/0 conn(0x7fb19c145590 legacy=0x7fb19c0b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19c0b3c80 con 0x7fb19c15f9b0 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fb190001670 con 0x7fb19c15f9b0 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 >> v1:172.21.15.77:6789/0 conn(0x7fb19c15f9b0 legacy=0x7fb19c06d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 --2- 172.21.15.77:0/2559664925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1840039b0 0x7fb184003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 --2- 172.21.15.77:0/2559664925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb184004800 0x7fb184004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 --2- 172.21.15.77:0/2559664925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1840051c0 0x7fb18400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb18400c030 con 0x7fb1840051c0 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb18400c1e0 con 0x7fb184004800 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb18400c3e0 con 0x7fb1840039b0 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.436+0000 7fb1a0ab3640 1 --2- 172.21.15.77:0/2559664925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1840051c0 0x7fb18400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.437+0000 7fb19b7fe640 1 --2- 172.21.15.77:0/2559664925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb184004800 0x7fb184004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.437+0000 7fb19bfff640 1 --2- 172.21.15.77:0/2559664925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1840039b0 0x7fb184003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.437+0000 7fb1a0ab3640 1 -- 172.21.15.77:0/2559664925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1840039b0 msgr2=0x7fb184003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:46.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.437+0000 7fb1a0ab3640 1 --2- 172.21.15.77:0/2559664925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1840039b0 0x7fb184003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.437+0000 7fb1a0ab3640 1 -- 172.21.15.77:0/2559664925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb184004800 msgr2=0x7fb184004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.437+0000 7fb1a0ab3640 1 --2- 172.21.15.77:0/2559664925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb184004800 0x7fb184004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.437+0000 7fb1a0ab3640 1 -- 172.21.15.77:0/2559664925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb184003520 con 0x7fb1840051c0 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.437+0000 7fb1a0ab3640 1 -- 172.21.15.77:0/2559664925 reap_dead start 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.437+0000 7fb1a0ab3640 1 --2- 172.21.15.77:0/2559664925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1840051c0 0x7fb18400b9c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb194009fc0 tx=0x7fb184005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87beb0265144b3c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.438+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194035020 con 0x7fb1840051c0 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.438+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb194003b90 con 0x7fb1840051c0 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.438+0000 7fb19affd640 1 -- 172.21.15.77:0/2559664925 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194009c10 con 0x7fb1840051c0 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.438+0000 7fb1a2516640 1 -- 172.21.15.77:0/2559664925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1840051c0 msgr2=0x7fb18400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.438+0000 7fb1a2516640 1 --2- 172.21.15.77:0/2559664925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1840051c0 0x7fb18400b9c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb194009fc0 tx=0x7fb184005830 comp rx=0 tx=0).stop 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.438+0000 7fb1a2516640 1 -- 172.21.15.77:0/2559664925 shutdown_connections 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.438+0000 7fb1a2516640 1 --2- 172.21.15.77:0/2559664925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1840051c0 0x7fb18400b9c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.438+0000 7fb1a2516640 1 -- 172.21.15.77:0/2559664925 >> 172.21.15.77:0/2559664925 conn(0x7fb19c0b1820 msgr2=0x7fb19c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:46.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.438+0000 7fb1a2516640 1 -- 172.21.15.77:0/2559664925 shutdown_connections 2024-04-25T01:22:46.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.438+0000 7fb1a2516640 1 -- 172.21.15.77:0/2559664925 wait complete. 2024-04-25T01:22:46.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.439+0000 7fb1a2516640 1 Processor -- start 2024-04-25T01:22:46.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.439+0000 7fb1a2516640 1 -- start start 2024-04-25T01:22:46.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb1a2516640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1840051c0 0x7fb19c13eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb1a2516640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb19c06e560 0x7fb19c071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb1a2516640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb19c06d170 0x7fb19c06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb1a2516640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb19c160280 con 0x7fb19c06e560 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb1a2516640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb19c06cde0 con 0x7fb19c06d170 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb1a2516640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb19c145970 con 0x7fb1840051c0 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb19b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb19c06e560 0x7fb19c071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb19b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb19c06e560 0x7fb19c071660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48406/0 (socket says 172.21.15.77:48406) 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb19b7fe640 1 -- 172.21.15.77:0/3886724949 learned_addr learned my addr 172.21.15.77:0/3886724949 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb19b7fe640 1 -- 172.21.15.77:0/3886724949 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1840051c0 msgr2=0x7fb19c13eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb19b7fe640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1840051c0 0x7fb19c13eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb19b7fe640 1 -- 172.21.15.77:0/3886724949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb19c06d170 msgr2=0x7fb19c06d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb1a0ab3640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb19c06d170 0x7fb19c06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb19b7fe640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb19c06d170 0x7fb19c06d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.440+0000 7fb19b7fe640 1 -- 172.21.15.77:0/3886724949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19c13f3f0 con 0x7fb19c06e560 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.441+0000 7fb19b7fe640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb19c06e560 0x7fb19c071660 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb190007340 tx=0x7fb190007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.441+0000 7fb198ff9640 1 -- 172.21.15.77:0/3886724949 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19000f8c0 con 0x7fb19c06e560 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.441+0000 7fb198ff9640 1 -- 172.21.15.77:0/3886724949 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb190018070 con 0x7fb19c06e560 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.441+0000 7fb1a2516640 1 -- 172.21.15.77:0/3886724949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb19c071e90 con 0x7fb19c06e560 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.441+0000 7fb198ff9640 1 -- 172.21.15.77:0/3886724949 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19000fca0 con 0x7fb19c06e560 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.441+0000 7fb1a2516640 1 -- 172.21.15.77:0/3886724949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb19c072450 con 0x7fb19c06e560 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.442+0000 7fb1a2516640 1 -- 172.21.15.77:0/3886724949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb19c068fb0 con 0x7fb19c06e560 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.442+0000 7fb198ff9640 1 -- 172.21.15.77:0/3886724949 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb19001d460 con 0x7fb19c06e560 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.443+0000 7fb198ff9640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb18806be30 0x7fb18806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.443+0000 7fb198ff9640 1 -- 172.21.15.77:0/3886724949 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6743+0+0 (secure 0 0 0) 0x7fb19008b2b0 con 0x7fb19c06e560 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.443+0000 7fb19bfff640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb18806be30 0x7fb18806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.443+0000 7fb19bfff640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb18806be30 0x7fb18806e2f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0016d0 tx=0x7fb18c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.447+0000 7fb198ff9640 1 -- 172.21.15.77:0/3886724949 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb1900598e0 con 0x7fb19c06e560 2024-04-25T01:22:46.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.654+0000 7fb1a2516640 1 -- 172.21.15.77:0/3886724949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb19c0af530 con 0x7fb19c06e560 2024-04-25T01:22:46.636 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-04-25T01:22:46.655+0000 7f46b1a53640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:22:46.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.658+0000 7fb198ff9640 1 -- 172.21.15.77:0/3886724949 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7fb1900591e0 con 0x7fb19c06e560 2024-04-25T01:22:46.639 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T01:22:46.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 -- 172.21.15.77:0/3886724949 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb18806be30 msgr2=0x7fb18806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb18806be30 0x7fb18806e2f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0016d0 tx=0x7fb18c009040 comp rx=0 tx=0).stop 2024-04-25T01:22:46.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 -- 172.21.15.77:0/3886724949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb19c06e560 msgr2=0x7fb19c071660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb19c06e560 0x7fb19c071660 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb190007340 tx=0x7fb190007800 comp rx=0 tx=0).stop 2024-04-25T01:22:46.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 -- 172.21.15.77:0/3886724949 shutdown_connections 2024-04-25T01:22:46.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb18806be30 0x7fb18806e2f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb19c06e560 0x7fb19c071660 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb19c06d170 0x7fb19c06d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 --2- 172.21.15.77:0/3886724949 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1840051c0 0x7fb19c13eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 -- 172.21.15.77:0/3886724949 >> 172.21.15.77:0/3886724949 conn(0x7fb19c0b1820 msgr2=0x7fb19c0afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:46.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 -- 172.21.15.77:0/3886724949 shutdown_connections 2024-04-25T01:22:46.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.660+0000 7fb1a2516640 1 -- 172.21.15.77:0/3886724949 wait complete. 2024-04-25T01:22:46.646 INFO:tasks.ceph.mds.d.smithi121.stderr: -5382> 2024-04-25T01:21:49.823+0000 7f279eae4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:22:46.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T01:22:46.679 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-25T01:22:46.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.771+0000 7f0546c46640 1 Processor -- start 2024-04-25T01:22:46.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.772+0000 7f0546c46640 1 -- start start 2024-04-25T01:22:46.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.772+0000 7f0546c46640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f05400b9a90 0x7f05400d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.772+0000 7f0546c46640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400d27b0 0x7f05400d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.772+0000 7f0546c46640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05400cea40 con 0x7f05400d23d0 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.772+0000 7f0546c46640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f05400cece0 con 0x7f05400b9a90 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.772+0000 7f0546c46640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f05400cefc0 con 0x7f05400d27b0 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.773+0000 7f05451e3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400d27b0 0x7f05400d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.773+0000 7f053ffff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f05400b9a90 0x7f05400d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.773+0000 7f05449e2640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f05400d23d0 0x7f05400b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:52614/0 (socket says 172.21.15.77:52614) 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.773+0000 7f05451e3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400d27b0 0x7f05400d0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:33406/0 (socket says 172.21.15.77:33406) 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.773+0000 7f05449e2640 1 -- 172.21.15.77:0/188341359 learned_addr learned my addr 172.21.15.77:0/188341359 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.773+0000 7f05451e3640 1 -- 172.21.15.77:0/188341359 >> v2:172.21.15.121:3300/0 conn(0x7f05400b9a90 msgr2=0x7f05400d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.773+0000 7f05451e3640 1 --2- 172.21.15.77:0/188341359 >> v2:172.21.15.121:3300/0 conn(0x7f05400b9a90 0x7f05400d4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.773+0000 7f05451e3640 1 -- 172.21.15.77:0/188341359 >> v1:172.21.15.121:6789/0 conn(0x7f05400d23d0 legacy=0x7f05400b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.773+0000 7f05451e3640 1 -- 172.21.15.77:0/188341359 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05400cf2a0 con 0x7f05400d27b0 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.774+0000 7f05451e3640 1 --2- 172.21.15.77:0/188341359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400d27b0 0x7f05400d0ee0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0538009d90 tx=0x7f053802a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec4d649ee8e0377c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.774+0000 7f053f7fe640 1 -- 172.21.15.77:0/188341359 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538007740 con 0x7f05400d27b0 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.774+0000 7f053f7fe640 1 -- 172.21.15.77:0/188341359 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05380078e0 con 0x7f05400d27b0 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.774+0000 7f053f7fe640 1 -- 172.21.15.77:0/188341359 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538007cb0 con 0x7f05400d27b0 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.775+0000 7f0546c46640 1 -- 172.21.15.77:0/188341359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400d27b0 msgr2=0x7f05400d0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.775+0000 7f0546c46640 1 --2- 172.21.15.77:0/188341359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400d27b0 0x7f05400d0ee0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0538009d90 tx=0x7f053802a6f0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.775+0000 7f0546c46640 1 -- 172.21.15.77:0/188341359 shutdown_connections 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.775+0000 7f0546c46640 1 --2- 172.21.15.77:0/188341359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400d27b0 0x7f05400d0ee0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.775+0000 7f0546c46640 1 --2- 172.21.15.77:0/188341359 >> v2:172.21.15.121:3300/0 conn(0x7f05400b9a90 0x7f05400d4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.775+0000 7f0546c46640 1 -- 172.21.15.77:0/188341359 >> 172.21.15.77:0/188341359 conn(0x7f05400d3e30 msgr2=0x7f05400d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.775+0000 7f0546c46640 1 -- 172.21.15.77:0/188341359 shutdown_connections 2024-04-25T01:22:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.775+0000 7f0546c46640 1 -- 172.21.15.77:0/188341359 wait complete. 2024-04-25T01:22:46.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.776+0000 7f0546c46640 1 Processor -- start 2024-04-25T01:22:46.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.776+0000 7f0546c46640 1 -- start start 2024-04-25T01:22:46.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f0546c46640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400b9a90 0x7f0540158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f0546c46640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400d23d0 0x7f054014b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f0546c46640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05400d27b0 0x7f05400c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f0546c46640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05400cfd80 con 0x7f05400d23d0 2024-04-25T01:22:46.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f0546c46640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f05400cf860 con 0x7f05400d27b0 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f0546c46640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f05400b6c80 con 0x7f05400b9a90 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400d23d0 0x7f054014b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400d23d0 0x7f054014b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48426/0 (socket says 172.21.15.77:48426) 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053ffff640 1 -- 172.21.15.77:0/1586206109 learned_addr learned my addr 172.21.15.77:0/1586206109 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f05449e2640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400b9a90 0x7f0540158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f05451e3640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05400d27b0 0x7f05400c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053ffff640 1 -- 172.21.15.77:0/1586206109 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400b9a90 msgr2=0x7f0540158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053ffff640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400b9a90 0x7f0540158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053ffff640 1 -- 172.21.15.77:0/1586206109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05400d27b0 msgr2=0x7f05400c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053ffff640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05400d27b0 0x7f05400c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053ffff640 1 -- 172.21.15.77:0/1586206109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05400ca4d0 con 0x7f05400d23d0 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f05449e2640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400b9a90 0x7f0540158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:46.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f05451e3640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05400d27b0 0x7f05400c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:46.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053ffff640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400d23d0 0x7f054014b2b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f052c00e9e0 tx=0x7f052c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053d7fa640 1 -- 172.21.15.77:0/1586206109 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c017070 con 0x7f05400d23d0 2024-04-25T01:22:46.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053d7fa640 1 -- 172.21.15.77:0/1586206109 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f052c00c9d0 con 0x7f05400d23d0 2024-04-25T01:22:46.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f0546c46640 1 -- 172.21.15.77:0/1586206109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0540158750 con 0x7f05400d23d0 2024-04-25T01:22:46.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.777+0000 7f053d7fa640 1 -- 172.21.15.77:0/1586206109 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c00cdc0 con 0x7f05400d23d0 2024-04-25T01:22:46.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.778+0000 7f0546c46640 1 -- 172.21.15.77:0/1586206109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0540158d10 con 0x7f05400d23d0 2024-04-25T01:22:46.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.778+0000 7f0546c46640 1 -- 172.21.15.77:0/1586206109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0540068fb0 con 0x7f05400d23d0 2024-04-25T01:22:46.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.779+0000 7f053d7fa640 1 -- 172.21.15.77:0/1586206109 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f052c01a070 con 0x7f05400d23d0 2024-04-25T01:22:46.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.779+0000 7f053d7fa640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f052406be30 0x7f052406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.779+0000 7f05449e2640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f052406be30 0x7f052406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.780+0000 7f053d7fa640 1 -- 172.21.15.77:0/1586206109 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6829+0+0 (secure 0 0 0) 0x7f052c08b2f0 con 0x7f05400d23d0 2024-04-25T01:22:46.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.780+0000 7f05449e2640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f052406be30 0x7f052406e2f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0534000f60 tx=0x7f0534009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.784+0000 7f053d7fa640 1 -- 172.21.15.77:0/1586206109 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f052c059840 con 0x7f05400d23d0 2024-04-25T01:22:46.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:46.992+0000 7f0546c46640 1 -- 172.21.15.77:0/1586206109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f054008df50 con 0x7f05400d23d0 2024-04-25T01:22:47.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.662+0000 7f053d7fa640 1 -- 172.21.15.77:0/1586206109 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f052c059140 con 0x7f05400d23d0 2024-04-25T01:22:47.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 -- 172.21.15.77:0/1586206109 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f052406be30 msgr2=0x7f052406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:47.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f052406be30 0x7f052406e2f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0534000f60 tx=0x7f0534009040 comp rx=0 tx=0).stop 2024-04-25T01:22:47.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 -- 172.21.15.77:0/1586206109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400d23d0 msgr2=0x7f054014b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:47.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400d23d0 0x7f054014b2b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f052c00e9e0 tx=0x7f052c00eea0 comp rx=0 tx=0).stop 2024-04-25T01:22:47.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 -- 172.21.15.77:0/1586206109 shutdown_connections 2024-04-25T01:22:47.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f052406be30 0x7f052406e2f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:47.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400d23d0 0x7f054014b2b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:47.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05400d27b0 0x7f05400c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:47.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 --2- 172.21.15.77:0/1586206109 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05400b9a90 0x7f0540158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:47.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 -- 172.21.15.77:0/1586206109 >> 172.21.15.77:0/1586206109 conn(0x7f05400d3e30 msgr2=0x7f05400b9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:47.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 -- 172.21.15.77:0/1586206109 shutdown_connections 2024-04-25T01:22:47.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.665+0000 7f0546c46640 1 -- 172.21.15.77:0/1586206109 wait complete. 2024-04-25T01:22:47.658 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T01:22:47.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.778+0000 7f11f9da9640 1 Processor -- start 2024-04-25T01:22:47.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.778+0000 7f11f9da9640 1 -- start start 2024-04-25T01:22:47.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.778+0000 7f11f9da9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11f415ae90 0x7f11f40ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:47.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.778+0000 7f11f9da9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f11f40ce470 0x7f11f40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:47.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.778+0000 7f11f9da9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11f4072480 con 0x7f11f406e2f0 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.779+0000 7f11f9da9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f11f40cf700 con 0x7f11f415ae90 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.779+0000 7f11f9da9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f11f40cd6b0 con 0x7f11f40ce470 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.779+0000 7f11f37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11f415ae90 0x7f11f40ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.779+0000 7f11f3fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f11f406e2f0 0x7f11f415e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36104/0 (socket says 172.21.15.77:36104) 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.779+0000 7f11f3fff640 1 -- 172.21.15.77:0/1587428027 learned_addr learned my addr 172.21.15.77:0/1587428027 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.779+0000 7f11ebfff640 1 --2- 172.21.15.77:0/1587428027 >> v2:172.21.15.121:3300/0 conn(0x7f11f40ce470 0x7f11f40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.779+0000 7f11f37fe640 1 -- 172.21.15.77:0/1587428027 >> v2:172.21.15.121:3300/0 conn(0x7f11f40ce470 msgr2=0x7f11f40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.779+0000 7f11f37fe640 1 --2- 172.21.15.77:0/1587428027 >> v2:172.21.15.121:3300/0 conn(0x7f11f40ce470 0x7f11f40cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.779+0000 7f11f37fe640 1 -- 172.21.15.77:0/1587428027 >> v1:172.21.15.77:6789/0 conn(0x7f11f406e2f0 legacy=0x7f11f415e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.779+0000 7f11f37fe640 1 -- 172.21.15.77:0/1587428027 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f40cd990 con 0x7f11f415ae90 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.779+0000 7f11f2ffd640 1 -- 172.21.15.77:0/1587428027 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3258735681 0 0) 0x7f11f4072480 con 0x7f11f406e2f0 2024-04-25T01:22:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f37fe640 1 --2- 172.21.15.77:0/1587428027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11f415ae90 0x7f11f40ceb90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f11f406c410 tx=0x7f11e402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf8bf6fbfe0c671 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:47.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f2ffd640 1 -- 172.21.15.77:0/1587428027 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4037070 con 0x7f11f415ae90 2024-04-25T01:22:47.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f2ffd640 1 -- 172.21.15.77:0/1587428027 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11e4032440 con 0x7f11f415ae90 2024-04-25T01:22:47.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f2ffd640 1 -- 172.21.15.77:0/1587428027 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4032810 con 0x7f11f415ae90 2024-04-25T01:22:47.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f9da9640 1 -- 172.21.15.77:0/1587428027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11f415ae90 msgr2=0x7f11f40ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f9da9640 1 --2- 172.21.15.77:0/1587428027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11f415ae90 0x7f11f40ceb90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f11f406c410 tx=0x7f11e402a1a0 comp rx=0 tx=0).stop 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f9da9640 1 -- 172.21.15.77:0/1587428027 shutdown_connections 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f9da9640 1 --2- 172.21.15.77:0/1587428027 >> v2:172.21.15.121:3300/0 conn(0x7f11f40ce470 0x7f11f40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f9da9640 1 --2- 172.21.15.77:0/1587428027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11f415ae90 0x7f11f40ceb90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f9da9640 1 -- 172.21.15.77:0/1587428027 >> 172.21.15.77:0/1587428027 conn(0x7f11f40cfae0 msgr2=0x7f11f40cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f9da9640 1 -- 172.21.15.77:0/1587428027 shutdown_connections 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.780+0000 7f11f9da9640 1 -- 172.21.15.77:0/1587428027 wait complete. 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.782+0000 7f11f9da9640 1 Processor -- start 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.782+0000 7f11f9da9640 1 -- start start 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.782+0000 7f11f9da9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11f406e2f0 0x7f11f414ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.782+0000 7f11f9da9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f40ce470 0x7f11f4143b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.782+0000 7f11f9da9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11f415ae90 0x7f11f40c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.782+0000 7f11f9da9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f11f406e7a0 con 0x7f11f406e2f0 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.782+0000 7f11f9da9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f11f40cdf50 con 0x7f11f415ae90 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.782+0000 7f11f9da9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f11f406e920 con 0x7f11f40ce470 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.782+0000 7f11f37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11f406e2f0 0x7f11f414ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.782+0000 7f11f37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11f406e2f0 0x7f11f414ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48448/0 (socket says 172.21.15.77:48448) 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.782+0000 7f11f37fe640 1 -- 172.21.15.77:0/2677284369 learned_addr learned my addr 172.21.15.77:0/2677284369 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f3fff640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11f415ae90 0x7f11f40c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11ebfff640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f40ce470 0x7f11f4143b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f37fe640 1 -- 172.21.15.77:0/2677284369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f40ce470 msgr2=0x7f11f4143b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f37fe640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f40ce470 0x7f11f4143b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f37fe640 1 -- 172.21.15.77:0/2677284369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11f415ae90 msgr2=0x7f11f40c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f37fe640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11f415ae90 0x7f11f40c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:47.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f37fe640 1 -- 172.21.15.77:0/2677284369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f40c9e00 con 0x7f11f406e2f0 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f37fe640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11f406e2f0 0x7f11f414ad40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f11e4035040 tx=0x7f11e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f0ff9640 1 -- 172.21.15.77:0/2677284369 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4037040 con 0x7f11f406e2f0 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f0ff9640 1 -- 172.21.15.77:0/2677284369 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11e4004230 con 0x7f11f406e2f0 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f9da9640 1 -- 172.21.15.77:0/2677284369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f414b320 con 0x7f11f406e2f0 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f0ff9640 1 -- 172.21.15.77:0/2677284369 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4003610 con 0x7f11f406e2f0 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.783+0000 7f11f9da9640 1 -- 172.21.15.77:0/2677284369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f4166df0 con 0x7f11f406e2f0 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.784+0000 7f11f9da9640 1 -- 172.21.15.77:0/2677284369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f4068fb0 con 0x7f11f406e2f0 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.784+0000 7f11f0ff9640 1 -- 172.21.15.77:0/2677284369 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f11e40037b0 con 0x7f11f406e2f0 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.785+0000 7f11f0ff9640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f11d406be30 0x7f11d406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.785+0000 7f11f0ff9640 1 -- 172.21.15.77:0/2677284369 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6829+0+0 (secure 0 0 0) 0x7f11e40ac3d0 con 0x7f11f406e2f0 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.785+0000 7f11ebfff640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f11d406be30 0x7f11d406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:47.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.785+0000 7f11ebfff640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f11d406be30 0x7f11d406e2f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f11f4158650 tx=0x7f11dc009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:47.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.789+0000 7f11f0ff9640 1 -- 172.21.15.77:0/2677284369 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f11e407a3f0 con 0x7f11f406e2f0 2024-04-25T01:22:47.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:47.962+0000 7f11f9da9640 1 -- 172.21.15.77:0/2677284369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f11f406ca90 con 0x7f11f406e2f0 2024-04-25T01:22:48.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.666+0000 7f11f0ff9640 1 -- 172.21.15.77:0/2677284369 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) ==== 181+0+0 (secure 0 0 0) 0x7f11e407e6f0 con 0x7f11f406e2f0 2024-04-25T01:22:48.648 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-04-25T01:22:48.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 -- 172.21.15.77:0/2677284369 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f11d406be30 msgr2=0x7f11d406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:48.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f11d406be30 0x7f11d406e2f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f11f4158650 tx=0x7f11dc009190 comp rx=0 tx=0).stop 2024-04-25T01:22:48.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 -- 172.21.15.77:0/2677284369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11f406e2f0 msgr2=0x7f11f414ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:48.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11f406e2f0 0x7f11f414ad40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f11e4035040 tx=0x7f11e40046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 -- 172.21.15.77:0/2677284369 shutdown_connections 2024-04-25T01:22:48.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f11d406be30 0x7f11d406e2f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11f415ae90 0x7f11f40c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f40ce470 0x7f11f4143b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 --2- 172.21.15.77:0/2677284369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11f406e2f0 0x7f11f414ad40 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 -- 172.21.15.77:0/2677284369 >> 172.21.15.77:0/2677284369 conn(0x7f11f40cfae0 msgr2=0x7f11f40cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:48.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 -- 172.21.15.77:0/2677284369 shutdown_connections 2024-04-25T01:22:48.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.668+0000 7f11f9da9640 1 -- 172.21.15.77:0/2677284369 wait complete. 2024-04-25T01:22:48.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T01:22:48.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.783+0000 7f16234ee640 1 Processor -- start 2024-04-25T01:22:48.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.784+0000 7f16234ee640 1 -- start start 2024-04-25T01:22:48.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.784+0000 7f16234ee640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f161c06cb40 0x7f161c070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:48.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.784+0000 7f16234ee640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f161c071110 0x7f161c0715a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:48.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.784+0000 7f16234ee640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f161c0722c0 con 0x7f161c15f870 2024-04-25T01:22:48.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.784+0000 7f16234ee640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f161c0725a0 con 0x7f161c071110 2024-04-25T01:22:48.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.784+0000 7f16234ee640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f161c0ca1d0 con 0x7f161c06cb40 2024-04-25T01:22:48.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.784+0000 7f1621a8b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f161c071110 0x7f161c0715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:48.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.784+0000 7f1621a8b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f161c071110 0x7f161c0715a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48454/0 (socket says 172.21.15.77:48454) 2024-04-25T01:22:48.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.785+0000 7f1621a8b640 1 -- 172.21.15.77:0/4083527621 learned_addr learned my addr 172.21.15.77:0/4083527621 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:48.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.785+0000 7f1620a89640 1 --2- 172.21.15.77:0/4083527621 >> v2:172.21.15.121:3300/0 conn(0x7f161c06cb40 0x7f161c070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.785+0000 7f1621a8b640 1 -- 172.21.15.77:0/4083527621 >> v2:172.21.15.121:3300/0 conn(0x7f161c06cb40 msgr2=0x7f161c070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.785+0000 7f1621a8b640 1 --2- 172.21.15.77:0/4083527621 >> v2:172.21.15.121:3300/0 conn(0x7f161c06cb40 0x7f161c070bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.785+0000 7f1621a8b640 1 -- 172.21.15.77:0/4083527621 >> v1:172.21.15.121:6789/0 conn(0x7f161c15f870 legacy=0x7f161c15fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.785+0000 7f1621a8b640 1 -- 172.21.15.77:0/4083527621 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161c0ca4b0 con 0x7f161c071110 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.785+0000 7f1621a8b640 1 --2- 172.21.15.77:0/4083527621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f161c071110 0x7f161c0715a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1614009e50 tx=0x7f161402a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7019996b80ea16ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.786+0000 7f1613fff640 1 -- 172.21.15.77:0/4083527621 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614036070 con 0x7f161c071110 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.786+0000 7f1613fff640 1 -- 172.21.15.77:0/4083527621 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1614031600 con 0x7f161c071110 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.786+0000 7f1613fff640 1 -- 172.21.15.77:0/4083527621 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16140319d0 con 0x7f161c071110 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.786+0000 7f16234ee640 1 -- 172.21.15.77:0/4083527621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f161c071110 msgr2=0x7f161c0715a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.786+0000 7f16234ee640 1 --2- 172.21.15.77:0/4083527621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f161c071110 0x7f161c0715a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1614009e50 tx=0x7f161402a760 comp rx=0 tx=0).stop 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.786+0000 7f16234ee640 1 -- 172.21.15.77:0/4083527621 shutdown_connections 2024-04-25T01:22:48.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.786+0000 7f16234ee640 1 --2- 172.21.15.77:0/4083527621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f161c071110 0x7f161c0715a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.786+0000 7f16234ee640 1 --2- 172.21.15.77:0/4083527621 >> v2:172.21.15.121:3300/0 conn(0x7f161c06cb40 0x7f161c070bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.786+0000 7f16234ee640 1 -- 172.21.15.77:0/4083527621 >> 172.21.15.77:0/4083527621 conn(0x7f161c0cfae0 msgr2=0x7f161c0cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:48.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.786+0000 7f16234ee640 1 -- 172.21.15.77:0/4083527621 shutdown_connections 2024-04-25T01:22:48.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.786+0000 7f16234ee640 1 -- 172.21.15.77:0/4083527621 wait complete. 2024-04-25T01:22:48.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.787+0000 7f16234ee640 1 Processor -- start 2024-04-25T01:22:48.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.787+0000 7f16234ee640 1 -- start start 2024-04-25T01:22:48.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f16234ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f161c06cb40 0x7f161c0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:48.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f16234ee640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f161c071110 0x7f161c1f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:48.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f16234ee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f161c15f870 0x7f161c0b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:48.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f16234ee640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f161c06cfa0 con 0x7f161c06cb40 2024-04-25T01:22:48.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f16234ee640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f161c145910 con 0x7f161c071110 2024-04-25T01:22:48.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f16234ee640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f161c0b4c10 con 0x7f161c15f870 2024-04-25T01:22:48.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f162128a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f161c06cb40 0x7f161c0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:48.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f162128a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f161c06cb40 0x7f161c0b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48464/0 (socket says 172.21.15.77:48464) 2024-04-25T01:22:48.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f162128a640 1 -- 172.21.15.77:0/958559375 learned_addr learned my addr 172.21.15.77:0/958559375 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f162128a640 1 -- 172.21.15.77:0/958559375 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f161c15f870 msgr2=0x7f161c0b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f162128a640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f161c15f870 0x7f161c0b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f162128a640 1 -- 172.21.15.77:0/958559375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f161c071110 msgr2=0x7f161c1f6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f162128a640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f161c071110 0x7f161c1f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f162128a640 1 -- 172.21.15.77:0/958559375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161c1f7390 con 0x7f161c06cb40 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.788+0000 7f162128a640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f161c06cb40 0x7f161c0b46d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f161800c3c0 tx=0x7f161800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.789+0000 7f1611ffb640 1 -- 172.21.15.77:0/958559375 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16180052d0 con 0x7f161c06cb40 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.789+0000 7f1611ffb640 1 -- 172.21.15.77:0/958559375 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1618005490 con 0x7f161c06cb40 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.789+0000 7f16234ee640 1 -- 172.21.15.77:0/958559375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f161c15d4a0 con 0x7f161c06cb40 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.789+0000 7f1611ffb640 1 -- 172.21.15.77:0/958559375 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1618007260 con 0x7f161c06cb40 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.789+0000 7f16234ee640 1 -- 172.21.15.77:0/958559375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f161c15dd40 con 0x7f161c06cb40 2024-04-25T01:22:48.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.790+0000 7f16234ee640 1 -- 172.21.15.77:0/958559375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f161c068fb0 con 0x7f161c06cb40 2024-04-25T01:22:48.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.790+0000 7f1611ffb640 1 -- 172.21.15.77:0/958559375 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1618007400 con 0x7f161c06cb40 2024-04-25T01:22:48.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.791+0000 7f1611ffb640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f160006be30 0x7f160006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:48.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.791+0000 7f1611ffb640 1 -- 172.21.15.77:0/958559375 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6391+0+0 (secure 0 0 0) 0x7f161808a560 con 0x7f161c06cb40 2024-04-25T01:22:48.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.791+0000 7f1620a89640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f160006be30 0x7f160006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:48.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.791+0000 7f1620a89640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f160006be30 0x7f160006e2f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f161c141c80 tx=0x7f160c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:48.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.795+0000 7f1611ffb640 1 -- 172.21.15.77:0/958559375 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1618058cf0 con 0x7f161c06cb40 2024-04-25T01:22:48.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:48.968+0000 7f16234ee640 1 -- 172.21.15.77:0/958559375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f161c15df10 con 0x7f161c06cb40 2024-04-25T01:22:49.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.674+0000 7f1611ffb640 1 -- 172.21.15.77:0/958559375 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) ==== 169+0+0 (secure 0 0 0) 0x7f16180585f0 con 0x7f161c06cb40 2024-04-25T01:22:49.656 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-04-25T01:22:49.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 -- 172.21.15.77:0/958559375 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f160006be30 msgr2=0x7f160006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:49.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f160006be30 0x7f160006e2f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f161c141c80 tx=0x7f160c008040 comp rx=0 tx=0).stop 2024-04-25T01:22:49.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 -- 172.21.15.77:0/958559375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f161c06cb40 msgr2=0x7f161c0b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:49.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f161c06cb40 0x7f161c0b46d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f161800c3c0 tx=0x7f161800c880 comp rx=0 tx=0).stop 2024-04-25T01:22:49.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 -- 172.21.15.77:0/958559375 shutdown_connections 2024-04-25T01:22:49.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f160006be30 0x7f160006e2f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f161c15f870 0x7f161c0b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f161c071110 0x7f161c1f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 --2- 172.21.15.77:0/958559375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f161c06cb40 0x7f161c0b46d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 -- 172.21.15.77:0/958559375 >> 172.21.15.77:0/958559375 conn(0x7f161c0cfae0 msgr2=0x7f161c0cb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:49.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 -- 172.21.15.77:0/958559375 shutdown_connections 2024-04-25T01:22:49.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.676+0000 7f16234ee640 1 -- 172.21.15.77:0/958559375 wait complete. 2024-04-25T01:22:49.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-25T01:22:49.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.796+0000 7f8d3d577640 1 Processor -- start 2024-04-25T01:22:49.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.796+0000 7f8d3d577640 1 -- start start 2024-04-25T01:22:49.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d3d577640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8d3815e4c0 0x7f8d380d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:49.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d3d577640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d380d4a10 0x7f8d380b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:49.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d3d577640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d380d1e90 con 0x7f8d380d2d40 2024-04-25T01:22:49.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d3d577640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f8d380b54a0 con 0x7f8d380d4a10 2024-04-25T01:22:49.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d3d577640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8d380d2130 con 0x7f8d3815e4c0 2024-04-25T01:22:49.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d367fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d380d4a10 0x7f8d380b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:49.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d367fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d380d4a10 0x7f8d380b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48470/0 (socket says 172.21.15.77:48470) 2024-04-25T01:22:49.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d367fc640 1 -- 172.21.15.77:0/3007645620 learned_addr learned my addr 172.21.15.77:0/3007645620 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:49.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d367fc640 1 -- 172.21.15.77:0/3007645620 >> v2:172.21.15.121:3300/0 conn(0x7f8d3815e4c0 msgr2=0x7f8d380d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:49.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d367fc640 1 --2- 172.21.15.77:0/3007645620 >> v2:172.21.15.121:3300/0 conn(0x7f8d3815e4c0 0x7f8d380d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d367fc640 1 -- 172.21.15.77:0/3007645620 >> v1:172.21.15.121:6789/0 conn(0x7f8d380d2d40 legacy=0x7f8d380d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:49.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.797+0000 7f8d367fc640 1 -- 172.21.15.77:0/3007645620 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d380b5780 con 0x7f8d380d4a10 2024-04-25T01:22:49.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d367fc640 1 --2- 172.21.15.77:0/3007645620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d380d4a10 0x7f8d380b3ee0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8d38067960 tx=0x7f8d2002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e5e158ccc3bbf56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:49.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d35ffb640 1 -- 172.21.15.77:0/3007645620 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20037070 con 0x7f8d380d4a10 2024-04-25T01:22:49.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d35ffb640 1 -- 172.21.15.77:0/3007645620 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d20032440 con 0x7f8d380d4a10 2024-04-25T01:22:49.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d35ffb640 1 -- 172.21.15.77:0/3007645620 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20032810 con 0x7f8d380d4a10 2024-04-25T01:22:49.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d3d577640 1 -- 172.21.15.77:0/3007645620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d380d4a10 msgr2=0x7f8d380b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:49.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d3d577640 1 --2- 172.21.15.77:0/3007645620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d380d4a10 0x7f8d380b3ee0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8d38067960 tx=0x7f8d2002a1a0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d3d577640 1 -- 172.21.15.77:0/3007645620 shutdown_connections 2024-04-25T01:22:49.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d3d577640 1 --2- 172.21.15.77:0/3007645620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d380d4a10 0x7f8d380b3ee0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d3d577640 1 --2- 172.21.15.77:0/3007645620 >> v2:172.21.15.121:3300/0 conn(0x7f8d3815e4c0 0x7f8d380d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d3d577640 1 -- 172.21.15.77:0/3007645620 >> 172.21.15.77:0/3007645620 conn(0x7f8d380d3ef0 msgr2=0x7f8d380d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:49.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d3d577640 1 -- 172.21.15.77:0/3007645620 shutdown_connections 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.798+0000 7f8d3d577640 1 -- 172.21.15.77:0/3007645620 wait complete. 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.800+0000 7f8d3d577640 1 Processor -- start 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.800+0000 7f8d3d577640 1 -- start start 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.800+0000 7f8d3d577640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d380d2d40 0x7f8d380c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.800+0000 7f8d3d577640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d380d4a10 0x7f8d380b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.800+0000 7f8d3d577640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d3815e4c0 0x7f8d381f95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.800+0000 7f8d3d577640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3815ea20 con 0x7f8d380d4a10 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.800+0000 7f8d3d577640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3815e8a0 con 0x7f8d3815e4c0 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.800+0000 7f8d3d577640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8d380d1010 con 0x7f8d380d2d40 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d380d4a10 0x7f8d380b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d380d4a10 0x7f8d380b8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48480/0 (socket says 172.21.15.77:48480) 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d367fc640 1 -- 172.21.15.77:0/365299774 learned_addr learned my addr 172.21.15.77:0/365299774 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d367fc640 1 -- 172.21.15.77:0/365299774 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d380d2d40 msgr2=0x7f8d380c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d367fc640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d380d2d40 0x7f8d380c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d367fc640 1 -- 172.21.15.77:0/365299774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d3815e4c0 msgr2=0x7f8d381f95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:49.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d377fe640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d3815e4c0 0x7f8d381f95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:49.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d367fc640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d3815e4c0 0x7f8d381f95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d367fc640 1 -- 172.21.15.77:0/365299774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d380c9350 con 0x7f8d380d4a10 2024-04-25T01:22:49.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d367fc640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d380d4a10 0x7f8d380b8c40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8d20000c00 tx=0x7f8d200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:49.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d2f7fe640 1 -- 172.21.15.77:0/365299774 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20041070 con 0x7f8d380d4a10 2024-04-25T01:22:49.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d2f7fe640 1 -- 172.21.15.77:0/365299774 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d2003c870 con 0x7f8d380d4a10 2024-04-25T01:22:49.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d2f7fe640 1 -- 172.21.15.77:0/365299774 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20037040 con 0x7f8d380d4a10 2024-04-25T01:22:49.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d3d577640 1 -- 172.21.15.77:0/365299774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d380c7d70 con 0x7f8d380d4a10 2024-04-25T01:22:49.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.801+0000 7f8d3d577640 1 -- 172.21.15.77:0/365299774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d380c8330 con 0x7f8d380d4a10 2024-04-25T01:22:49.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.802+0000 7f8d3d577640 1 -- 172.21.15.77:0/365299774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d38068fb0 con 0x7f8d380d4a10 2024-04-25T01:22:49.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.803+0000 7f8d2f7fe640 1 -- 172.21.15.77:0/365299774 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8d20044070 con 0x7f8d380d4a10 2024-04-25T01:22:49.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.803+0000 7f8d2f7fe640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d1806be30 0x7f8d1806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:49.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.803+0000 7f8d36ffd640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d1806be30 0x7f8d1806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:49.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.803+0000 7f8d2f7fe640 1 -- 172.21.15.77:0/365299774 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5853+0+0 (secure 0 0 0) 0x7f8d200abc80 con 0x7f8d380d4a10 2024-04-25T01:22:49.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.804+0000 7f8d36ffd640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d1806be30 0x7f8d1806e2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800ab90 tx=0x7f8d28009250 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:49.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.808+0000 7f8d2f7fe640 1 -- 172.21.15.77:0/365299774 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8d2007a750 con 0x7f8d380d4a10 2024-04-25T01:22:49.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:49.981+0000 7f8d3d577640 1 -- 172.21.15.77:0/365299774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8d380d4520 con 0x7f8d380d4a10 2024-04-25T01:22:50.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.679+0000 7f8d2f7fe640 1 -- 172.21.15.77:0/365299774 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) ==== 178+0+0 (secure 0 0 0) 0x7f8d2007a050 con 0x7f8d380d4a10 2024-04-25T01:22:50.661 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data_ec' removed 2024-04-25T01:22:50.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.682+0000 7f8d3d577640 1 -- 172.21.15.77:0/365299774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d1806be30 msgr2=0x7f8d1806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:50.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.682+0000 7f8d3d577640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d1806be30 0x7f8d1806e2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800ab90 tx=0x7f8d28009250 comp rx=0 tx=0).stop 2024-04-25T01:22:50.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.682+0000 7f8d3d577640 1 -- 172.21.15.77:0/365299774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d380d4a10 msgr2=0x7f8d380b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:50.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.682+0000 7f8d3d577640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d380d4a10 0x7f8d380b8c40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8d20000c00 tx=0x7f8d200046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:50.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.683+0000 7f8d3d577640 1 -- 172.21.15.77:0/365299774 shutdown_connections 2024-04-25T01:22:50.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.683+0000 7f8d3d577640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d1806be30 0x7f8d1806e2f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:50.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.683+0000 7f8d3d577640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d380d4a10 0x7f8d380b8c40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:50.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.683+0000 7f8d3d577640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d3815e4c0 0x7f8d381f95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:50.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.683+0000 7f8d3d577640 1 --2- 172.21.15.77:0/365299774 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d380d2d40 0x7f8d380c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:50.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.683+0000 7f8d3d577640 1 -- 172.21.15.77:0/365299774 >> 172.21.15.77:0/365299774 conn(0x7f8d380d3ef0 msgr2=0x7f8d3806cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:50.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.683+0000 7f8d3d577640 1 -- 172.21.15.77:0/365299774 shutdown_connections 2024-04-25T01:22:50.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.683+0000 7f8d3d577640 1 -- 172.21.15.77:0/365299774 wait complete. 2024-04-25T01:22:50.677 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T01:22:50.677 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T01:22:50.677 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T01:22:50.678 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:22:50.678 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T01:22:50.678 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T01:22:50.678 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T01:22:50.679 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:22:50.679 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T01:22:50.679 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T01:22:50.679 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T01:22:50.679 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:22:50.679 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T01:22:50.679 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T01:22:50.681 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T01:22:50.681 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:22:50.687 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T01:22:50.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T01:22:50.688 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T01:22:50.689 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T01:22:50.691 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T01:22:50.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T01:22:50.693 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T01:22:50.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T01:22:50.729 INFO:tasks.ceph.mds.a:Started 2024-04-25T01:22:50.730 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:50.730 INFO:tasks.ceph.mds.c:Started 2024-04-25T01:22:50.731 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:50.734 INFO:tasks.ceph.mds.b:Started 2024-04-25T01:22:50.734 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:50.735 INFO:tasks.ceph.mds.d:Started 2024-04-25T01:22:50.735 DEBUG:teuthology.parallel:result is None 2024-04-25T01:22:50.735 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T01:22:50.810 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-04-25T01:22:50.811 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-04-25T01:22:50.819 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-25T01:22:50.820 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-25T01:22:50.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.864+0000 7ff4f07fb640 1 Processor -- start 2024-04-25T01:22:50.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.864+0000 7ff4f07fb640 1 -- start start 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4f07fb640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff4e815e4c0 0x7ff4e80d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4f07fb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff4e80d4a10 0x7ff4e80b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4f07fb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4e80d1e90 con 0x7ff4e80d2d40 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4f07fb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7ff4e80b54a0 con 0x7ff4e80d4a10 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4f07fb640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff4e80d2130 con 0x7ff4e815e4c0 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4edd96640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff4e80d4a10 0x7ff4e80b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4edd96640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff4e80d4a10 0x7ff4e80b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48534/0 (socket says 172.21.15.77:48534) 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4edd96640 1 -- 172.21.15.77:0/862842456 learned_addr learned my addr 172.21.15.77:0/862842456 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4eed98640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff4e80d2d40 0x7ff4e80d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36120/0 (socket says 172.21.15.77:36120) 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4ee597640 1 --2- 172.21.15.77:0/862842456 >> v2:172.21.15.121:3300/0 conn(0x7ff4e815e4c0 0x7ff4e80d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4edd96640 1 -- 172.21.15.77:0/862842456 >> v2:172.21.15.121:3300/0 conn(0x7ff4e815e4c0 msgr2=0x7ff4e80d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4edd96640 1 --2- 172.21.15.77:0/862842456 >> v2:172.21.15.121:3300/0 conn(0x7ff4e815e4c0 0x7ff4e80d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4ed595640 1 -- 172.21.15.77:0/862842456 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3589456396 0 0) 0x7ff4e80d1e90 con 0x7ff4e80d2d40 2024-04-25T01:22:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.865+0000 7ff4edd96640 1 -- 172.21.15.77:0/862842456 >> v1:172.21.15.77:6789/0 conn(0x7ff4e80d2d40 legacy=0x7ff4e80d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.866+0000 7ff4edd96640 1 -- 172.21.15.77:0/862842456 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e80b5780 con 0x7ff4e80d4a10 2024-04-25T01:22:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.866+0000 7ff4ee597640 1 --2- 172.21.15.77:0/862842456 >> v2:172.21.15.121:3300/0 conn(0x7ff4e815e4c0 0x7ff4e80d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.866+0000 7ff4edd96640 1 --2- 172.21.15.77:0/862842456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4e80d4a10 0x7ff4e80b3ee0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8067960 tx=0x7ff4d802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1289e2daeaf1f8f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.866+0000 7ff4ed595640 1 -- 172.21.15.77:0/862842456 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8037070 con 0x7ff4e80d4a10 2024-04-25T01:22:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.866+0000 7ff4ed595640 1 -- 172.21.15.77:0/862842456 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4d8032440 con 0x7ff4e80d4a10 2024-04-25T01:22:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.866+0000 7ff4ed595640 1 -- 172.21.15.77:0/862842456 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8032810 con 0x7ff4e80d4a10 2024-04-25T01:22:50.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.867+0000 7ff4f07fb640 1 -- 172.21.15.77:0/862842456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4e80d4a10 msgr2=0x7ff4e80b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:50.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.867+0000 7ff4f07fb640 1 --2- 172.21.15.77:0/862842456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4e80d4a10 0x7ff4e80b3ee0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8067960 tx=0x7ff4d802a200 comp rx=0 tx=0).stop 2024-04-25T01:22:50.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.867+0000 7ff4f07fb640 1 -- 172.21.15.77:0/862842456 shutdown_connections 2024-04-25T01:22:50.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.867+0000 7ff4f07fb640 1 --2- 172.21.15.77:0/862842456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4e80d4a10 0x7ff4e80b3ee0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:50.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.867+0000 7ff4f07fb640 1 --2- 172.21.15.77:0/862842456 >> v2:172.21.15.121:3300/0 conn(0x7ff4e815e4c0 0x7ff4e80d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:50.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.867+0000 7ff4f07fb640 1 -- 172.21.15.77:0/862842456 >> 172.21.15.77:0/862842456 conn(0x7ff4e80d3ef0 msgr2=0x7ff4e80d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:50.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.867+0000 7ff4f07fb640 1 -- 172.21.15.77:0/862842456 shutdown_connections 2024-04-25T01:22:50.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.867+0000 7ff4f07fb640 1 -- 172.21.15.77:0/862842456 wait complete. 2024-04-25T01:22:50.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.868+0000 7ff4f07fb640 1 Processor -- start 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.868+0000 7ff4f07fb640 1 -- start start 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4f07fb640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80d2d40 0x7ff4e80cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4f07fb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4e80d4a10 0x7ff4e814aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4f07fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4e815e4c0 0x7ff4e80ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4f07fb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e815ea20 con 0x7ff4e815e4c0 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4f07fb640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e815e8a0 con 0x7ff4e80d2d40 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4f07fb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff4e80d1010 con 0x7ff4e80d4a10 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4eed98640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4e815e4c0 0x7ff4e80ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4eed98640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4e815e4c0 0x7ff4e80ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48540/0 (socket says 172.21.15.77:48540) 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4eed98640 1 -- 172.21.15.77:0/839578641 learned_addr learned my addr 172.21.15.77:0/839578641 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4ee597640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80d2d40 0x7ff4e80cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4edd96640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4e80d4a10 0x7ff4e814aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:50.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4eed98640 1 -- 172.21.15.77:0/839578641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4e80d4a10 msgr2=0x7ff4e814aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4eed98640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4e80d4a10 0x7ff4e814aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4eed98640 1 -- 172.21.15.77:0/839578641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80d2d40 msgr2=0x7ff4e80cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4eed98640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80d2d40 0x7ff4e80cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4eed98640 1 -- 172.21.15.77:0/839578641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e814e3a0 con 0x7ff4e815e4c0 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.869+0000 7ff4eed98640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4e815e4c0 0x7ff4e80ceaa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c7a0 tx=0x7ff4e000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.870+0000 7ff4deffd640 1 -- 172.21.15.77:0/839578641 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0018070 con 0x7ff4e815e4c0 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.870+0000 7ff4f07fb640 1 -- 172.21.15.77:0/839578641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4e80cd040 con 0x7ff4e815e4c0 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.870+0000 7ff4deffd640 1 -- 172.21.15.77:0/839578641 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4e000a640 con 0x7ff4e815e4c0 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.870+0000 7ff4deffd640 1 -- 172.21.15.77:0/839578641 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e000aab0 con 0x7ff4e815e4c0 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.870+0000 7ff4f07fb640 1 -- 172.21.15.77:0/839578641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4e80cd600 con 0x7ff4e815e4c0 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.871+0000 7ff4f07fb640 1 -- 172.21.15.77:0/839578641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4e8068fb0 con 0x7ff4e815e4c0 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.871+0000 7ff4deffd640 1 -- 172.21.15.77:0/839578641 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff4e000ac50 con 0x7ff4e815e4c0 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.872+0000 7ff4deffd640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff4cc06be30 0x7ff4cc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.872+0000 7ff4deffd640 1 -- 172.21.15.77:0/839578641 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5335+0+0 (secure 0 0 0) 0x7ff4e008a360 con 0x7ff4e815e4c0 2024-04-25T01:22:50.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.872+0000 7ff4ee597640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff4cc06be30 0x7ff4cc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:50.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.872+0000 7ff4ee597640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff4cc06be30 0x7ff4cc06e2f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4006fd0 tx=0x7ff4e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:50.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:50.876+0000 7ff4deffd640 1 -- 172.21.15.77:0/839578641 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff4e0058e90 con 0x7ff4e815e4c0 2024-04-25T01:22:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.061+0000 7ff4f07fb640 1 -- 172.21.15.77:0/839578641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ff4e806bd50 con 0x7ff4e815e4c0 2024-04-25T01:22:51.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.693+0000 7ff4deffd640 1 -- 172.21.15.77:0/839578641 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) ==== 97+0+0 (secure 0 0 0) 0x7ff4e0088e20 con 0x7ff4e815e4c0 2024-04-25T01:22:51.675 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-04-25T01:22:51.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.696+0000 7ff4f07fb640 1 -- 172.21.15.77:0/839578641 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff4cc06be30 msgr2=0x7ff4cc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:51.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.696+0000 7ff4f07fb640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff4cc06be30 0x7ff4cc06e2f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4006fd0 tx=0x7ff4e4008040 comp rx=0 tx=0).stop 2024-04-25T01:22:51.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.696+0000 7ff4f07fb640 1 -- 172.21.15.77:0/839578641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4e815e4c0 msgr2=0x7ff4e80ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:51.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.696+0000 7ff4f07fb640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4e815e4c0 0x7ff4e80ceaa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c7a0 tx=0x7ff4e000cc60 comp rx=0 tx=0).stop 2024-04-25T01:22:51.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.696+0000 7ff4f07fb640 1 -- 172.21.15.77:0/839578641 shutdown_connections 2024-04-25T01:22:51.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.696+0000 7ff4f07fb640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff4cc06be30 0x7ff4cc06e2f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.696+0000 7ff4f07fb640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4e815e4c0 0x7ff4e80ceaa0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.696+0000 7ff4f07fb640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4e80d4a10 0x7ff4e814aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.696+0000 7ff4f07fb640 1 --2- 172.21.15.77:0/839578641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80d2d40 0x7ff4e80cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.697+0000 7ff4f07fb640 1 -- 172.21.15.77:0/839578641 >> 172.21.15.77:0/839578641 conn(0x7ff4e80d3ef0 msgr2=0x7ff4e80d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:51.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.697+0000 7ff4f07fb640 1 -- 172.21.15.77:0/839578641 shutdown_connections 2024-04-25T01:22:51.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.697+0000 7ff4f07fb640 1 -- 172.21.15.77:0/839578641 wait complete. 2024-04-25T01:22:51.692 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T01:22:51.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.810+0000 7ff249ebd640 1 Processor -- start 2024-04-25T01:22:51.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.810+0000 7ff249ebd640 1 -- start start 2024-04-25T01:22:51.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff249ebd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff244145590 0x7ff2440b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff249ebd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff24406e280 0x7ff24415fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff249ebd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2440b4240 con 0x7ff24415f9b0 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff249ebd640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7ff244160830 con 0x7ff24406e280 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff249ebd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff244160350 con 0x7ff244145590 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff243fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff24406e280 0x7ff24415fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff243fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff24406e280 0x7ff24415fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48552/0 (socket says 172.21.15.77:48552) 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff243fff640 1 -- 172.21.15.77:0/1266727572 learned_addr learned my addr 172.21.15.77:0/1266727572 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff243fff640 1 -- 172.21.15.77:0/1266727572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff244145590 msgr2=0x7ff2440b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff2437fe640 1 --2- 172.21.15.77:0/1266727572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff244145590 0x7ff2440b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff243fff640 1 --2- 172.21.15.77:0/1266727572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff244145590 0x7ff2440b4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff243fff640 1 -- 172.21.15.77:0/1266727572 >> v1:172.21.15.121:6789/0 conn(0x7ff24415f9b0 legacy=0x7ff2440ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:51.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff243fff640 1 -- 172.21.15.77:0/1266727572 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2440b4f60 con 0x7ff24406e280 2024-04-25T01:22:51.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.811+0000 7ff2437fe640 1 --2- 172.21.15.77:0/1266727572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff244145590 0x7ff2440b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:51.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.812+0000 7ff243fff640 1 --2- 172.21.15.77:0/1266727572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff24406e280 0x7ff24415fe10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009e50 tx=0x7ff23c02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fde603b8186e7b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:51.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.812+0000 7ff2427fc640 1 -- 172.21.15.77:0/1266727572 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c036070 con 0x7ff24406e280 2024-04-25T01:22:51.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.812+0000 7ff2427fc640 1 -- 172.21.15.77:0/1266727572 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff23c031600 con 0x7ff24406e280 2024-04-25T01:22:51.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.812+0000 7ff2427fc640 1 -- 172.21.15.77:0/1266727572 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c0319d0 con 0x7ff24406e280 2024-04-25T01:22:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.813+0000 7ff249ebd640 1 -- 172.21.15.77:0/1266727572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff24406e280 msgr2=0x7ff24415fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.813+0000 7ff249ebd640 1 --2- 172.21.15.77:0/1266727572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff24406e280 0x7ff24415fe10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009e50 tx=0x7ff23c02a6c0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.813+0000 7ff249ebd640 1 -- 172.21.15.77:0/1266727572 shutdown_connections 2024-04-25T01:22:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.813+0000 7ff249ebd640 1 --2- 172.21.15.77:0/1266727572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff24406e280 0x7ff24415fe10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.813+0000 7ff249ebd640 1 --2- 172.21.15.77:0/1266727572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff244145590 0x7ff2440b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.813+0000 7ff249ebd640 1 -- 172.21.15.77:0/1266727572 >> 172.21.15.77:0/1266727572 conn(0x7ff2440b1820 msgr2=0x7ff2440b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.813+0000 7ff249ebd640 1 -- 172.21.15.77:0/1266727572 shutdown_connections 2024-04-25T01:22:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.813+0000 7ff249ebd640 1 -- 172.21.15.77:0/1266727572 wait complete. 2024-04-25T01:22:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.814+0000 7ff249ebd640 1 Processor -- start 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.814+0000 7ff249ebd640 1 -- start start 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff249ebd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff24406e280 0x7ff2441e7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff249ebd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff244145590 0x7ff2441f63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff249ebd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff24415f9b0 0x7ff24415e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff249ebd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff2440b3c80 con 0x7ff24406e280 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff249ebd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff24406cde0 con 0x7ff24415f9b0 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff249ebd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff2440b3f60 con 0x7ff244145590 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff2437fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff24406e280 0x7ff2441e7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff2437fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff24406e280 0x7ff2441e7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48568/0 (socket says 172.21.15.77:48568) 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff2437fe640 1 -- 172.21.15.77:0/2950522824 learned_addr learned my addr 172.21.15.77:0/2950522824 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff242ffd640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff244145590 0x7ff2441f63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff2437fe640 1 -- 172.21.15.77:0/2950522824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff244145590 msgr2=0x7ff2441f63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:51.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff2437fe640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff244145590 0x7ff2441f63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff2437fe640 1 -- 172.21.15.77:0/2950522824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff24415f9b0 msgr2=0x7ff24415e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff2437fe640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff24415f9b0 0x7ff24415e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff2437fe640 1 -- 172.21.15.77:0/2950522824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff244158240 con 0x7ff24406e280 2024-04-25T01:22:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff2437fe640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff24406e280 0x7ff2441e7040 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff23400c7f0 tx=0x7ff23400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff22bfff640 1 -- 172.21.15.77:0/2950522824 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff234015070 con 0x7ff24406e280 2024-04-25T01:22:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff22bfff640 1 -- 172.21.15.77:0/2950522824 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff234007820 con 0x7ff24406e280 2024-04-25T01:22:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff249ebd640 1 -- 172.21.15.77:0/2950522824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff244158520 con 0x7ff24406e280 2024-04-25T01:22:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.815+0000 7ff22bfff640 1 -- 172.21.15.77:0/2950522824 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff234007bd0 con 0x7ff24406e280 2024-04-25T01:22:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.816+0000 7ff249ebd640 1 -- 172.21.15.77:0/2950522824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff244131c90 con 0x7ff24406e280 2024-04-25T01:22:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.816+0000 7ff249ebd640 1 -- 172.21.15.77:0/2950522824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2440cce90 con 0x7ff24406e280 2024-04-25T01:22:51.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.817+0000 7ff22bfff640 1 -- 172.21.15.77:0/2950522824 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff234018070 con 0x7ff24406e280 2024-04-25T01:22:51.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.817+0000 7ff22bfff640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff23006be30 0x7ff23006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:51.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.817+0000 7ff22bfff640 1 -- 172.21.15.77:0/2950522824 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7ff23408ad30 con 0x7ff24406e280 2024-04-25T01:22:51.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.817+0000 7ff242ffd640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff23006be30 0x7ff23006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:51.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.818+0000 7ff242ffd640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff23006be30 0x7ff23006e2f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff2441f5920 tx=0x7ff238009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:51.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:51.822+0000 7ff22bfff640 1 -- 172.21.15.77:0/2950522824 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff2440cce90 con 0x7ff24406e280 2024-04-25T01:22:52.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.062+0000 7ff249ebd640 1 -- 172.21.15.77:0/2950522824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7ff244068fb0 con 0x7ff24406e280 2024-04-25T01:22:52.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.063+0000 7ff22bfff640 1 -- 172.21.15.77:0/2950522824 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7ff234059930 con 0x7ff24406e280 2024-04-25T01:22:52.044 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmRBubJBAA1jgiWa9DTdamG6ZHPTzJHA==", 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmxEjKLxAAaPXHEf3AJlrRvnyH4OKCpg==", 2024-04-25T01:22:52.045 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmc1hYKxAATtGeOU4Ep9JGOclelSbShw==", 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-04-25T01:22:52.046 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylm8tWDNhAAq3irRf+ZuOWsrOLP59WOuA==", 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-04-25T01:22:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDzrylmFiLEFxAAiu3CMixQObEnTzVB4+oIOg==", 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQD2rylm0cMJFhAAVluT5EojBn9sI3GaJjJRqw==", 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQD5rylmklH8ExAASXBlrt1JxxxhCynyR08Isw==", 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQD8rylmKQ3jERAARBac0vky2rIDzUdr4nYn/A==", 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.049 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQACsClmaGfoGBAAtbFXxGHH/zXHHDB3UBzlTA==", 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.050 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQAFsClm7GCeFhAANCOnPFNJhLmUfK+Kk+zHUw==", 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQAIsClmC2hZFBAAEem007DX6DrvGHwN5YTjgw==", 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:22:52.051 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQALsClmijV3FBAANujp5DQhJDc8IYocyllBBA==", 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmsfhcOhAAYZeVeMoyKmZdON0jUeFmeQ==", 2024-04-25T01:22:52.052 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDwrylm8sQzBxAAJNZ00VUA4tzDCB+UZb+gnQ==", 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.053 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDwrylmRdgcBBAA4/WiRMEX2/s9wbevmTPgLA==", 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDwrylmG5KlDBAA28sSTbOA+nZ1ejbMP6Mqig==", 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:22:52.054 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmuucRChAAIjlRw0RadOzdjfDM5tigrQ==", 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-mds", 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARsClmGaDpBRAA1JOLUnZKeYPb1KcxdV95nQ==", 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-mds" 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-mgr", 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARsClmoarpBRAAN3Y/9/wwisXbb67alU1Vbg==", 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-osd", 2024-04-25T01:22:52.056 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARsClmxLHpBRAAmbosSVWPVVk6Da3rhUv2BA==", 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-osd" 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rbd", 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARsClmULjpBRAAdiVk/DcfeZvnbsxC2TSmeQ==", 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARsClm+r7pBRAAr6eSgwXLib4xCP5frU0YAQ==", 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.057 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rgw", 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARsClmj8fpBRAAnqOXfufxLp33b6bIxlkfKQ==", 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmw1qCIRAAqlQEapgQsaTD1GhzKlJX/A==", 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-04-25T01:22:52.058 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmqtQqHhAAaEIOEzQbQjkAPt2EIRAixQ==", 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-04-25T01:22:52.059 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 -- 172.21.15.77:0/2950522824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff23006be30 msgr2=0x7ff23006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff23006be30 0x7ff23006e2f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff2441f5920 tx=0x7ff238009040 comp rx=0 tx=0).stop 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 -- 172.21.15.77:0/2950522824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff24406e280 msgr2=0x7ff2441e7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff24406e280 0x7ff2441e7040 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff23400c7f0 tx=0x7ff23400ccb0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 -- 172.21.15.77:0/2950522824 shutdown_connections 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff23006be30 0x7ff23006e2f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff24415f9b0 0x7ff24415e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff244145590 0x7ff2441f63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 --2- 172.21.15.77:0/2950522824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff24406e280 0x7ff2441e7040 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 -- 172.21.15.77:0/2950522824 >> 172.21.15.77:0/2950522824 conn(0x7ff2440b1820 msgr2=0x7ff2440b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 -- 172.21.15.77:0/2950522824 shutdown_connections 2024-04-25T01:22:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.065+0000 7ff249ebd640 1 -- 172.21.15.77:0/2950522824 wait complete. 2024-04-25T01:22:52.062 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-25T01:22:52.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.180+0000 7f470bfff640 1 Processor -- start 2024-04-25T01:22:52.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.180+0000 7f470bfff640 1 -- start start 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f470c15a380 0x7f470c15e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f470c0b4590 0x7f470c070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470bfff640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c072930 con 0x7f470c0aea00 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470bfff640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f470c06e3a0 con 0x7f470c15a380 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470bfff640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f470c06e030 con 0x7f470c0b4590 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470affd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f470c15a380 0x7f470c15e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f4703fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f470c0aea00 0x7f470c0b3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36122/0 (socket says 172.21.15.77:36122) 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f4703fff640 1 -- 172.21.15.77:0/438074634 learned_addr learned my addr 172.21.15.77:0/438074634 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470b7fe640 1 --2- 172.21.15.77:0/438074634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f470c0b4590 0x7f470c070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470affd640 1 -- 172.21.15.77:0/438074634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f470c0b4590 msgr2=0x7f470c070a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470affd640 1 --2- 172.21.15.77:0/438074634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f470c0b4590 0x7f470c070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470affd640 1 -- 172.21.15.77:0/438074634 >> v1:172.21.15.77:6789/0 conn(0x7f470c0aea00 legacy=0x7f470c0b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470a7fc640 1 -- 172.21.15.77:0/438074634 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2134981371 0 0) 0x7f470c072930 con 0x7f470c0aea00 2024-04-25T01:22:52.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.181+0000 7f470affd640 1 -- 172.21.15.77:0/438074634 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c15a760 con 0x7f470c15a380 2024-04-25T01:22:52.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.182+0000 7f470affd640 1 --2- 172.21.15.77:0/438074634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470c15a380 0x7f470c15e970 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0097a0 tx=0x7f46fc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cffad0bf775eb56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:52.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.182+0000 7f470a7fc640 1 -- 172.21.15.77:0/438074634 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc037070 con 0x7f470c15a380 2024-04-25T01:22:52.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.182+0000 7f470a7fc640 1 -- 172.21.15.77:0/438074634 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46fc032440 con 0x7f470c15a380 2024-04-25T01:22:52.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.182+0000 7f470a7fc640 1 -- 172.21.15.77:0/438074634 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc032810 con 0x7f470c15a380 2024-04-25T01:22:52.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.183+0000 7f470bfff640 1 -- 172.21.15.77:0/438074634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470c15a380 msgr2=0x7f470c15e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.183+0000 7f470bfff640 1 --2- 172.21.15.77:0/438074634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470c15a380 0x7f470c15e970 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0097a0 tx=0x7f46fc02a180 comp rx=0 tx=0).stop 2024-04-25T01:22:52.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.183+0000 7f470bfff640 1 -- 172.21.15.77:0/438074634 shutdown_connections 2024-04-25T01:22:52.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.183+0000 7f470bfff640 1 --2- 172.21.15.77:0/438074634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f470c0b4590 0x7f470c070a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.183+0000 7f470bfff640 1 --2- 172.21.15.77:0/438074634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470c15a380 0x7f470c15e970 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.183+0000 7f470bfff640 1 -- 172.21.15.77:0/438074634 >> 172.21.15.77:0/438074634 conn(0x7f470c0b1660 msgr2=0x7f470c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:52.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.183+0000 7f470bfff640 1 -- 172.21.15.77:0/438074634 shutdown_connections 2024-04-25T01:22:52.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.183+0000 7f470bfff640 1 -- 172.21.15.77:0/438074634 wait complete. 2024-04-25T01:22:52.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.184+0000 7f470bfff640 1 Processor -- start 2024-04-25T01:22:52.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.184+0000 7f470bfff640 1 -- start start 2024-04-25T01:22:52.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f470bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f470c0aea00 0x7f470c141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f470bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470c0b4590 0x7f470c1f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f470bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f470c15a380 0x7f470c1f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f470bfff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f470c06b120 con 0x7f470c0b4590 2024-04-25T01:22:52.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f470bfff640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f470c0af150 con 0x7f470c0aea00 2024-04-25T01:22:52.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f470bfff640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f470c06b2a0 con 0x7f470c15a380 2024-04-25T01:22:52.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f4703fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470c0b4590 0x7f470c1f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f4703fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470c0b4590 0x7f470c1f34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48574/0 (socket says 172.21.15.77:48574) 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f4703fff640 1 -- 172.21.15.77:0/2986219356 learned_addr learned my addr 172.21.15.77:0/2986219356 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f470b7fe640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f470c15a380 0x7f470c1f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f470affd640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f470c0aea00 0x7f470c141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f4703fff640 1 -- 172.21.15.77:0/2986219356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f470c15a380 msgr2=0x7f470c1f3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f4703fff640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f470c15a380 0x7f470c1f3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f4703fff640 1 -- 172.21.15.77:0/2986219356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f470c0aea00 msgr2=0x7f470c141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f4703fff640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f470c0aea00 0x7f470c141320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f4703fff640 1 -- 172.21.15.77:0/2986219356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c159200 con 0x7f470c0b4590 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f470affd640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f470c0aea00 0x7f470c141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f470b7fe640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f470c15a380 0x7f470c1f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f4703fff640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470c0b4590 0x7f470c1f34d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f46f400c4d0 tx=0x7f46f400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f47037fe640 1 -- 172.21.15.77:0/2986219356 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f4018070 con 0x7f470c0b4590 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f47037fe640 1 -- 172.21.15.77:0/2986219356 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46f4007500 con 0x7f470c0b4590 2024-04-25T01:22:52.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f47037fe640 1 -- 172.21.15.77:0/2986219356 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f4004a20 con 0x7f470c0b4590 2024-04-25T01:22:52.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.185+0000 7f470bfff640 1 -- 172.21.15.77:0/2986219356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470c1594e0 con 0x7f470c0b4590 2024-04-25T01:22:52.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.186+0000 7f470bfff640 1 -- 172.21.15.77:0/2986219356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470c150dd0 con 0x7f470c0b4590 2024-04-25T01:22:52.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.186+0000 7f470bfff640 1 -- 172.21.15.77:0/2986219356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470c0b1f40 con 0x7f470c0b4590 2024-04-25T01:22:52.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.187+0000 7f47037fe640 1 -- 172.21.15.77:0/2986219356 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f46f401b070 con 0x7f470c0b4590 2024-04-25T01:22:52.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.187+0000 7f47037fe640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f46ec06be30 0x7f46ec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.187+0000 7f470affd640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f46ec06be30 0x7f46ec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.187+0000 7f47037fe640 1 -- 172.21.15.77:0/2986219356 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f46f408a020 con 0x7f470c0b4590 2024-04-25T01:22:52.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.188+0000 7f470affd640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f46ec06be30 0x7f46ec06e2f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f46fc034000 tx=0x7f46fc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.192+0000 7f47037fe640 1 -- 172.21.15.77:0/2986219356 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f470c0b1f40 con 0x7f470c0b4590 2024-04-25T01:22:52.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.431+0000 7f470bfff640 1 -- 172.21.15.77:0/2986219356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f470c075350 con 0x7f470c0b4590 2024-04-25T01:22:52.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.432+0000 7f47037fe640 1 -- 172.21.15.77:0/2986219356 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f46f4058b90 con 0x7f470c0b4590 2024-04-25T01:22:52.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.434+0000 7f470bfff640 1 -- 172.21.15.77:0/2986219356 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f46ec06be30 msgr2=0x7f46ec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.434+0000 7f470bfff640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f46ec06be30 0x7f46ec06e2f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f46fc034000 tx=0x7f46fc02e000 comp rx=0 tx=0).stop 2024-04-25T01:22:52.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.434+0000 7f470bfff640 1 -- 172.21.15.77:0/2986219356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470c0b4590 msgr2=0x7f470c1f34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.434+0000 7f470bfff640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470c0b4590 0x7f470c1f34d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f46f400c4d0 tx=0x7f46f400c990 comp rx=0 tx=0).stop 2024-04-25T01:22:52.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.434+0000 7f470bfff640 1 -- 172.21.15.77:0/2986219356 shutdown_connections 2024-04-25T01:22:52.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.434+0000 7f470bfff640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f46ec06be30 0x7f46ec06e2f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.434+0000 7f470bfff640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470c0b4590 0x7f470c1f34d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.434+0000 7f470bfff640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f470c15a380 0x7f470c1f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.434+0000 7f470bfff640 1 --2- 172.21.15.77:0/2986219356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f470c0aea00 0x7f470c141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.434+0000 7f470bfff640 1 -- 172.21.15.77:0/2986219356 >> 172.21.15.77:0/2986219356 conn(0x7f470c0b1660 msgr2=0x7f470c155450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:52.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.435+0000 7f470bfff640 1 -- 172.21.15.77:0/2986219356 shutdown_connections 2024-04-25T01:22:52.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.435+0000 7f470bfff640 1 -- 172.21.15.77:0/2986219356 wait complete. 2024-04-25T01:22:52.427 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-25T01:22:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.542+0000 7f6b750f5640 1 Processor -- start 2024-04-25T01:22:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.543+0000 7f6b750f5640 1 -- start start 2024-04-25T01:22:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.543+0000 7f6b750f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7006ed60 0x7f6b7015f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.543+0000 7f6b750f5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b70070e60 con 0x7f6b70143940 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.543+0000 7f6b750f5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b70071140 con 0x7f6b70143d20 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.543+0000 7f6b750f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6b70071420 con 0x7f6b7006ed60 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.543+0000 7f6b6ed9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6b70143940 0x7f6b7006e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36134/0 (socket says 172.21.15.77:36134) 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6ed9d640 1 -- 172.21.15.77:0/4089291542 learned_addr learned my addr 172.21.15.77:0/4089291542 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 61160737 0 0) 0x7f6b70070e60 con 0x7f6b70143940 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b58003520 con 0x7f6b70143940 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6f59e640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7006ed60 0x7f6b7015f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2114460197 0 0) 0x7f6b58003520 con 0x7f6b70143940 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7006ed60 msgr2=0x7f6b7015f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7006ed60 0x7f6b7015f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 >> v1:172.21.15.121:6789/0 conn(0x7f6b70143d20 legacy=0x7f6b7015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b70071700 con 0x7f6b70143940 2024-04-25T01:22:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f6b64001670 con 0x7f6b70143940 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 >> v1:172.21.15.77:6789/0 conn(0x7f6b70143940 legacy=0x7f6b7006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7006ed60 0x7f6b580039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b58003f60 0x7f6b58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b58004d40 0x7f6b580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b70071420 con 0x7f6b58004d40 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5800baa0 con 0x7f6b58003f60 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6b5800deb0 con 0x7f6b7006ed60 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6f59e640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b58004d40 0x7f6b580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6ed9d640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7006ed60 0x7f6b580039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6e59c640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b58003f60 0x7f6b58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6f59e640 1 -- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7006ed60 msgr2=0x7f6b580039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6f59e640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7006ed60 0x7f6b580039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6f59e640 1 -- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b58003f60 msgr2=0x7f6b58004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.544+0000 7f6b6f59e640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b58003f60 0x7f6b58004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.545+0000 7f6b6f59e640 1 -- 172.21.15.77:0/4089291542 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b58003520 con 0x7f6b58004d40 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.545+0000 7f6b6ed9d640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7006ed60 0x7f6b580039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.545+0000 7f6b6f59e640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b58004d40 0x7f6b580051c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009f90 tx=0x7f6b58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d2b440f5292e21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.545+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b68009c10 con 0x7f6b58004d40 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.545+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b68036070 con 0x7f6b58004d40 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.545+0000 7f6b6dd9b640 1 -- 172.21.15.77:0/4089291542 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b68031850 con 0x7f6b58004d40 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.546+0000 7f6b750f5640 1 -- 172.21.15.77:0/4089291542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b58004d40 msgr2=0x7f6b580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.546+0000 7f6b750f5640 1 --2- 172.21.15.77:0/4089291542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b58004d40 0x7f6b580051c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009f90 tx=0x7f6b58005830 comp rx=0 tx=0).stop 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.546+0000 7f6b6f59e640 1 -- 172.21.15.77:0/4089291542 reap_dead start 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.546+0000 7f6b750f5640 1 -- 172.21.15.77:0/4089291542 shutdown_connections 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.546+0000 7f6b750f5640 1 -- 172.21.15.77:0/4089291542 >> 172.21.15.77:0/4089291542 conn(0x7f6b700b1820 msgr2=0x7f6b700b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.546+0000 7f6b750f5640 1 -- 172.21.15.77:0/4089291542 shutdown_connections 2024-04-25T01:22:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.546+0000 7f6b750f5640 1 -- 172.21.15.77:0/4089291542 wait complete. 2024-04-25T01:22:52.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.547+0000 7f6b750f5640 1 Processor -- start 2024-04-25T01:22:52.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.547+0000 7f6b750f5640 1 -- start start 2024-04-25T01:22:52.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b750f5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b701555b0 0x7f6b7015aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b750f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b70158a80 0x7f6b7015caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b750f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7015d030 0x7f6b70157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b750f5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7015f400 con 0x7f6b70158a80 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b750f5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6b70072e40 con 0x7f6b701555b0 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b750f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6b7015f580 con 0x7f6b7015d030 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b70158a80 0x7f6b7015caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b70158a80 0x7f6b7015caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48592/0 (socket says 172.21.15.77:48592) 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6e59c640 1 -- 172.21.15.77:0/3474625925 learned_addr learned my addr 172.21.15.77:0/3474625925 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6ed9d640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b701555b0 0x7f6b7015aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6f59e640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7015d030 0x7f6b70157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6e59c640 1 -- 172.21.15.77:0/3474625925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7015d030 msgr2=0x7f6b70157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6e59c640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7015d030 0x7f6b70157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6e59c640 1 -- 172.21.15.77:0/3474625925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b701555b0 msgr2=0x7f6b7015aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6e59c640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b701555b0 0x7f6b7015aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6e59c640 1 -- 172.21.15.77:0/3474625925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b70158f80 con 0x7f6b70158a80 2024-04-25T01:22:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6ed9d640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b701555b0 0x7f6b7015aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.548+0000 7f6b6e59c640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b70158a80 0x7f6b7015caf0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000f620 tx=0x7f6b6000fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.549+0000 7f6b577fe640 1 -- 172.21.15.77:0/3474625925 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b600052e0 con 0x7f6b70158a80 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.549+0000 7f6b577fe640 1 -- 172.21.15.77:0/3474625925 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b60018070 con 0x7f6b70158a80 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.549+0000 7f6b750f5640 1 -- 172.21.15.77:0/3474625925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b70159260 con 0x7f6b70158a80 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.549+0000 7f6b577fe640 1 -- 172.21.15.77:0/3474625925 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b60003810 con 0x7f6b70158a80 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.549+0000 7f6b750f5640 1 -- 172.21.15.77:0/3474625925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b70159820 con 0x7f6b70158a80 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.550+0000 7f6b750f5640 1 -- 172.21.15.77:0/3474625925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b70068fb0 con 0x7f6b70158a80 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.550+0000 7f6b577fe640 1 -- 172.21.15.77:0/3474625925 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6b600039b0 con 0x7f6b70158a80 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.550+0000 7f6b577fe640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b5c06be60 0x7f6b5c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.550+0000 7f6b577fe640 1 -- 172.21.15.77:0/3474625925 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f6b6008aa60 con 0x7f6b70158a80 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.551+0000 7f6b6ed9d640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b5c06be60 0x7f6b5c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.551+0000 7f6b6ed9d640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b5c06be60 0x7f6b5c06e320 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6b64000fd0 tx=0x7f6b64001ad0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:52.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.555+0000 7f6b577fe640 1 -- 172.21.15.77:0/3474625925 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6b60059540 con 0x7f6b70158a80 2024-04-25T01:22:52.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.799+0000 7f6b750f5640 1 -- 172.21.15.77:0/3474625925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f6b7006d150 con 0x7f6b70158a80 2024-04-25T01:22:52.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.801+0000 7f6b577fe640 1 -- 172.21.15.77:0/3474625925 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f6b60058e40 con 0x7f6b70158a80 2024-04-25T01:22:52.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 -- 172.21.15.77:0/3474625925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b5c06be60 msgr2=0x7f6b5c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b5c06be60 0x7f6b5c06e320 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6b64000fd0 tx=0x7f6b64001ad0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 -- 172.21.15.77:0/3474625925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b70158a80 msgr2=0x7f6b7015caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b70158a80 0x7f6b7015caf0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000f620 tx=0x7f6b6000fae0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 -- 172.21.15.77:0/3474625925 shutdown_connections 2024-04-25T01:22:52.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b5c06be60 0x7f6b5c06e320 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b70158a80 0x7f6b7015caf0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b7015d030 0x7f6b70157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 --2- 172.21.15.77:0/3474625925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b701555b0 0x7f6b7015aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 -- 172.21.15.77:0/3474625925 >> 172.21.15.77:0/3474625925 conn(0x7f6b700b1820 msgr2=0x7f6b700af770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:52.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 -- 172.21.15.77:0/3474625925 shutdown_connections 2024-04-25T01:22:52.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.803+0000 7f6b750f5640 1 -- 172.21.15.77:0/3474625925 wait complete. 2024-04-25T01:22:52.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-25T01:22:52.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.911+0000 7f807f514640 1 Processor -- start 2024-04-25T01:22:52.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.911+0000 7f807f514640 1 -- start start 2024-04-25T01:22:52.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807f514640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f807806fcd0 0x7f807815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807f514640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f807806c840 0x7f80780b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807f514640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80780b3560 con 0x7f80780ade10 2024-04-25T01:22:52.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807f514640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f80781589e0 con 0x7f807806fcd0 2024-04-25T01:22:52.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807f514640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f80780b3890 con 0x7f807806c840 2024-04-25T01:22:52.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807d2b0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f807806fcd0 0x7f807815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807caaf640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f80780ade10 0x7f807806c130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36138/0 (socket says 172.21.15.77:36138) 2024-04-25T01:22:52.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807caaf640 1 -- 172.21.15.77:0/1244224476 learned_addr learned my addr 172.21.15.77:0/1244224476 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:52.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807dab1640 1 --2- 172.21.15.77:0/1244224476 >> v2:172.21.15.121:3300/0 conn(0x7f807806c840 0x7f80780b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807d2b0640 1 -- 172.21.15.77:0/1244224476 >> v2:172.21.15.121:3300/0 conn(0x7f807806c840 msgr2=0x7f80780b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:52.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807d2b0640 1 --2- 172.21.15.77:0/1244224476 >> v2:172.21.15.121:3300/0 conn(0x7f807806c840 0x7f80780b3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.912+0000 7f807d2b0640 1 -- 172.21.15.77:0/1244224476 >> v1:172.21.15.77:6789/0 conn(0x7f80780ade10 legacy=0x7f807806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.913+0000 7f807d2b0640 1 -- 172.21.15.77:0/1244224476 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80780ae510 con 0x7f807806fcd0 2024-04-25T01:22:52.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.913+0000 7f806ffff640 1 -- 172.21.15.77:0/1244224476 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 353109481 0 0) 0x7f80780b3560 con 0x7f80780ade10 2024-04-25T01:22:52.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.913+0000 7f807d2b0640 1 --2- 172.21.15.77:0/1244224476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f807806fcd0 0x7f807815ef50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f807815f5a0 tx=0x7f807402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69a28c3ab6edd27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:52.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.913+0000 7f806ffff640 1 -- 172.21.15.77:0/1244224476 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074007740 con 0x7f807806fcd0 2024-04-25T01:22:52.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.913+0000 7f806ffff640 1 -- 172.21.15.77:0/1244224476 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80740078e0 con 0x7f807806fcd0 2024-04-25T01:22:52.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.913+0000 7f806ffff640 1 -- 172.21.15.77:0/1244224476 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074007cb0 con 0x7f807806fcd0 2024-04-25T01:22:52.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.914+0000 7f807f514640 1 -- 172.21.15.77:0/1244224476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f807806fcd0 msgr2=0x7f807815ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.914+0000 7f807f514640 1 --2- 172.21.15.77:0/1244224476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f807806fcd0 0x7f807815ef50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f807815f5a0 tx=0x7f807402a5f0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.914+0000 7f807f514640 1 -- 172.21.15.77:0/1244224476 shutdown_connections 2024-04-25T01:22:52.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.914+0000 7f807f514640 1 --2- 172.21.15.77:0/1244224476 >> v2:172.21.15.121:3300/0 conn(0x7f807806c840 0x7f80780b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.914+0000 7f807f514640 1 --2- 172.21.15.77:0/1244224476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f807806fcd0 0x7f807815ef50 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.914+0000 7f807f514640 1 -- 172.21.15.77:0/1244224476 >> 172.21.15.77:0/1244224476 conn(0x7f8078071550 msgr2=0x7f8078071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:52.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.914+0000 7f807f514640 1 -- 172.21.15.77:0/1244224476 shutdown_connections 2024-04-25T01:22:52.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.914+0000 7f807f514640 1 -- 172.21.15.77:0/1244224476 wait complete. 2024-04-25T01:22:52.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.915+0000 7f807f514640 1 Processor -- start 2024-04-25T01:22:52.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.915+0000 7f807f514640 1 -- start start 2024-04-25T01:22:52.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807f514640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f807806c840 0x7f80781f1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807f514640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f807806fcd0 0x7f80781f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807f514640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80780ade10 0x7f807815a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807f514640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f80780aead0 con 0x7f80780ade10 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807f514640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f80780ae2c0 con 0x7f807806c840 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807f514640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f80780b4260 con 0x7f807806fcd0 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807dab1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80780ade10 0x7f807815a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807dab1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80780ade10 0x7f807815a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48600/0 (socket says 172.21.15.77:48600) 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807dab1640 1 -- 172.21.15.77:0/1726813980 learned_addr learned my addr 172.21.15.77:0/1726813980 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807d2b0640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f807806c840 0x7f80781f1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807caaf640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f807806fcd0 0x7f80781f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807dab1640 1 -- 172.21.15.77:0/1726813980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f807806fcd0 msgr2=0x7f80781f4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807dab1640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f807806fcd0 0x7f80781f4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807dab1640 1 -- 172.21.15.77:0/1726813980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f807806c840 msgr2=0x7f80781f1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:52.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807dab1640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f807806c840 0x7f80781f1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:52.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807dab1640 1 -- 172.21.15.77:0/1726813980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8078159120 con 0x7f80780ade10 2024-04-25T01:22:52.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807d2b0640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f807806c840 0x7f80781f1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:52.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f807dab1640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80780ade10 0x7f807815a070 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f807000d600 tx=0x7f807000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:52.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.916+0000 7f806dffb640 1 -- 172.21.15.77:0/1726813980 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80700125f0 con 0x7f80780ade10 2024-04-25T01:22:52.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.917+0000 7f806dffb640 1 -- 172.21.15.77:0/1726813980 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8070019070 con 0x7f80780ade10 2024-04-25T01:22:52.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.917+0000 7f807f514640 1 -- 172.21.15.77:0/1726813980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8078159400 con 0x7f80780ade10 2024-04-25T01:22:52.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.917+0000 7f806dffb640 1 -- 172.21.15.77:0/1726813980 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80700129d0 con 0x7f80780ade10 2024-04-25T01:22:52.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.917+0000 7f807f514640 1 -- 172.21.15.77:0/1726813980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80781599c0 con 0x7f80780ade10 2024-04-25T01:22:52.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.918+0000 7f807f514640 1 -- 172.21.15.77:0/1726813980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8078068fb0 con 0x7f80780ade10 2024-04-25T01:22:52.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.918+0000 7f806dffb640 1 -- 172.21.15.77:0/1726813980 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8070012b70 con 0x7f80780ade10 2024-04-25T01:22:52.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.918+0000 7f806dffb640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f805c06be30 0x7f805c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:52.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.919+0000 7f806dffb640 1 -- 172.21.15.77:0/1726813980 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f807008b150 con 0x7f80780ade10 2024-04-25T01:22:52.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.919+0000 7f807d2b0640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f805c06be30 0x7f805c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:52.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.919+0000 7f807d2b0640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f805c06be30 0x7f805c06e2f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8074034000 tx=0x7f807402f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:52.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:52.923+0000 7f806dffb640 1 -- 172.21.15.77:0/1726813980 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8070059cc0 con 0x7f80780ade10 2024-04-25T01:22:53.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.161+0000 7f807f514640 1 -- 172.21.15.77:0/1726813980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f80780cf5e0 con 0x7f80780ade10 2024-04-25T01:22:53.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.163+0000 7f806dffb640 1 -- 172.21.15.77:0/1726813980 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f80700595c0 con 0x7f80780ade10 2024-04-25T01:22:53.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 -- 172.21.15.77:0/1726813980 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f805c06be30 msgr2=0x7f805c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f805c06be30 0x7f805c06e2f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8074034000 tx=0x7f807402f000 comp rx=0 tx=0).stop 2024-04-25T01:22:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 -- 172.21.15.77:0/1726813980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80780ade10 msgr2=0x7f807815a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80780ade10 0x7f807815a070 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f807000d600 tx=0x7f807000dac0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 -- 172.21.15.77:0/1726813980 shutdown_connections 2024-04-25T01:22:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f805c06be30 0x7f805c06e2f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80780ade10 0x7f807815a070 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f807806fcd0 0x7f80781f4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 --2- 172.21.15.77:0/1726813980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f807806c840 0x7f80781f1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 -- 172.21.15.77:0/1726813980 >> 172.21.15.77:0/1726813980 conn(0x7f8078071550 msgr2=0x7f8078151970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 -- 172.21.15.77:0/1726813980 shutdown_connections 2024-04-25T01:22:53.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.166+0000 7f807f514640 1 -- 172.21.15.77:0/1726813980 wait complete. 2024-04-25T01:22:53.158 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-25T01:22:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.273+0000 7f6c03f5a640 1 Processor -- start 2024-04-25T01:22:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.273+0000 7f6c03f5a640 1 -- start start 2024-04-25T01:22:53.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.274+0000 7f6c03f5a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6bfc15e4c0 0x7f6bfc0d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.274+0000 7f6c03f5a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0d2d40 0x7f6bfc0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.274+0000 7f6c03f5a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bfc0d1e90 con 0x7f6bfc0d4a10 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.274+0000 7f6c03f5a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6bfc0b54a0 con 0x7f6bfc15e4c0 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.274+0000 7f6c03f5a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6bfc0d2130 con 0x7f6bfc0d2d40 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.274+0000 7f6c014f5640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6bfc0d4a10 0x7f6bfc0b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36140/0 (socket says 172.21.15.77:36140) 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.274+0000 7f6c014f5640 1 -- 172.21.15.77:0/365435996 learned_addr learned my addr 172.21.15.77:0/365435996 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1912778375 0 0) 0x7f6bfc0d1e90 con 0x7f6bfc0d4a10 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6be8003520 con 0x7f6bfc0d4a10 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c024f7640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0d2d40 0x7f6bfc0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c01cf6640 1 --2- 172.21.15.77:0/365435996 >> v2:172.21.15.121:3300/0 conn(0x7f6bfc15e4c0 0x7f6bfc0d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4208133775 0 0) 0x7f6be8003520 con 0x7f6bfc0d4a10 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0d2d40 msgr2=0x7f6bfc0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0d2d40 0x7f6bfc0cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 >> v2:172.21.15.121:3300/0 conn(0x7f6bfc15e4c0 msgr2=0x7f6bfc0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 --2- 172.21.15.77:0/365435996 >> v2:172.21.15.121:3300/0 conn(0x7f6bfc15e4c0 0x7f6bfc0d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc0b5780 con 0x7f6bfc0d4a10 2024-04-25T01:22:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f6bec001670 con 0x7f6bfc0d4a10 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c01cf6640 1 --2- 172.21.15.77:0/365435996 >> v2:172.21.15.121:3300/0 conn(0x7f6bfc15e4c0 0x7f6bfc0d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 >> v1:172.21.15.77:6789/0 conn(0x7f6bfc0d4a10 legacy=0x7f6bfc0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0d2d40 0x7f6be80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 0x7f6be8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6be8004d40 0x7f6be80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc0b54a0 con 0x7f6be8003fe0 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc0d2130 con 0x7f6be8004d40 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6be800bb80 con 0x7f6bfc0d2d40 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c014f5640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 0x7f6be8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c01cf6640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0d2d40 0x7f6be80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c024f7640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6be8004d40 0x7f6be80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c014f5640 1 -- 172.21.15.77:0/365435996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0d2d40 msgr2=0x7f6be80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c014f5640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0d2d40 0x7f6be80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c014f5640 1 -- 172.21.15.77:0/365435996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6be8004d40 msgr2=0x7f6be80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c014f5640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6be8004d40 0x7f6be80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.275+0000 7f6c014f5640 1 -- 172.21.15.77:0/365435996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be8003520 con 0x7f6be8003fe0 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.276+0000 7f6c014f5640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 0x7f6be8004800 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6bec001050 tx=0x7f6be8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bff5ac3037d3e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.276+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec007d70 con 0x7f6be8003fe0 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.276+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bec005af0 con 0x7f6be8003fe0 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.276+0000 7f6c00cf4640 1 -- 172.21.15.77:0/365435996 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec007070 con 0x7f6be8003fe0 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.277+0000 7f6c03f5a640 1 -- 172.21.15.77:0/365435996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 msgr2=0x7f6be8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.277+0000 7f6c03f5a640 1 --2- 172.21.15.77:0/365435996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 0x7f6be8004800 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6bec001050 tx=0x7f6be8005830 comp rx=0 tx=0).stop 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.277+0000 7f6c024f7640 1 -- 172.21.15.77:0/365435996 reap_dead start 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.277+0000 7f6c03f5a640 1 -- 172.21.15.77:0/365435996 shutdown_connections 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.277+0000 7f6c03f5a640 1 -- 172.21.15.77:0/365435996 >> 172.21.15.77:0/365435996 conn(0x7f6bfc0d3ef0 msgr2=0x7f6bfc0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.277+0000 7f6c03f5a640 1 -- 172.21.15.77:0/365435996 shutdown_connections 2024-04-25T01:22:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.277+0000 7f6c03f5a640 1 -- 172.21.15.77:0/365435996 wait complete. 2024-04-25T01:22:53.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.278+0000 7f6c03f5a640 1 Processor -- start 2024-04-25T01:22:53.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.278+0000 7f6c03f5a640 1 -- start start 2024-04-25T01:22:53.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c03f5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 0x7f6bfc0cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c03f5a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0cb420 0x7f6bfc0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c03f5a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bfc0cbdc0 0x7f6bfc0b84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c03f5a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc15ea20 con 0x7f6be8003fe0 2024-04-25T01:22:53.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c03f5a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc15e8a0 con 0x7f6bfc0cbdc0 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c03f5a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6bfc0b45f0 con 0x7f6bfc0cb420 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c01cf6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 0x7f6bfc0cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c01cf6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 0x7f6bfc0cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48616/0 (socket says 172.21.15.77:48616) 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c01cf6640 1 -- 172.21.15.77:0/4269266709 learned_addr learned my addr 172.21.15.77:0/4269266709 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c01cf6640 1 -- 172.21.15.77:0/4269266709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0cb420 msgr2=0x7f6bfc0cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c014f5640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0cb420 0x7f6bfc0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c024f7640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bfc0cbdc0 0x7f6bfc0b84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c01cf6640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0cb420 0x7f6bfc0cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c01cf6640 1 -- 172.21.15.77:0/4269266709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bfc0cbdc0 msgr2=0x7f6bfc0b84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c01cf6640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bfc0cbdc0 0x7f6bfc0b84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c01cf6640 1 -- 172.21.15.77:0/4269266709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc0cc3c0 con 0x7f6be8003fe0 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c024f7640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bfc0cbdc0 0x7f6bfc0b84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c014f5640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0cb420 0x7f6bfc0cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c01cf6640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 0x7f6bfc0cdd60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8010190 tx=0x7f6bf8010650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6bf27fc640 1 -- 172.21.15.77:0/4269266709 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8010e80 con 0x7f6be8003fe0 2024-04-25T01:22:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6bf27fc640 1 -- 172.21.15.77:0/4269266709 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bf8003990 con 0x7f6be8003fe0 2024-04-25T01:22:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6bf27fc640 1 -- 172.21.15.77:0/4269266709 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8003b40 con 0x7f6be8003fe0 2024-04-25T01:22:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c03f5a640 1 -- 172.21.15.77:0/4269266709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bfc0b1630 con 0x7f6be8003fe0 2024-04-25T01:22:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.279+0000 7f6c03f5a640 1 -- 172.21.15.77:0/4269266709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bfc0b1bf0 con 0x7f6be8003fe0 2024-04-25T01:22:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.280+0000 7f6c03f5a640 1 -- 172.21.15.77:0/4269266709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bfc068fb0 con 0x7f6be8003fe0 2024-04-25T01:22:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.281+0000 7f6bf27fc640 1 -- 172.21.15.77:0/4269266709 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6bf801e020 con 0x7f6be8003fe0 2024-04-25T01:22:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.281+0000 7f6bf27fc640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6be006be60 0x7f6be006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.281+0000 7f6bf27fc640 1 -- 172.21.15.77:0/4269266709 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f6bf808b680 con 0x7f6be8003fe0 2024-04-25T01:22:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.281+0000 7f6c014f5640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6be006be60 0x7f6be006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.282+0000 7f6c014f5640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6be006be60 0x7f6be006e320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6bec033000 tx=0x7f6bec009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.285+0000 7f6bf27fc640 1 -- 172.21.15.77:0/4269266709 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6bf8025020 con 0x7f6be8003fe0 2024-04-25T01:22:53.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.523+0000 7f6c03f5a640 1 -- 172.21.15.77:0/4269266709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f6bfc06d640 con 0x7f6be8003fe0 2024-04-25T01:22:53.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.525+0000 7f6bf27fc640 1 -- 172.21.15.77:0/4269266709 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f6bf805a1f0 con 0x7f6be8003fe0 2024-04-25T01:22:53.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 -- 172.21.15.77:0/4269266709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6be006be60 msgr2=0x7f6be006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6be006be60 0x7f6be006e320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6bec033000 tx=0x7f6bec009030 comp rx=0 tx=0).stop 2024-04-25T01:22:53.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 -- 172.21.15.77:0/4269266709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 msgr2=0x7f6bfc0cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 0x7f6bfc0cdd60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8010190 tx=0x7f6bf8010650 comp rx=0 tx=0).stop 2024-04-25T01:22:53.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 -- 172.21.15.77:0/4269266709 shutdown_connections 2024-04-25T01:22:53.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6be006be60 0x7f6be006e320 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bfc0cbdc0 0x7f6bfc0b84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bfc0cb420 0x7f6bfc0cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 --2- 172.21.15.77:0/4269266709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be8003fe0 0x7f6bfc0cdd60 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 -- 172.21.15.77:0/4269266709 >> 172.21.15.77:0/4269266709 conn(0x7f6bfc0d3ef0 msgr2=0x7f6bfc0b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 -- 172.21.15.77:0/4269266709 shutdown_connections 2024-04-25T01:22:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.527+0000 7f6c03f5a640 1 -- 172.21.15.77:0/4269266709 wait complete. 2024-04-25T01:22:53.519 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-25T01:22:53.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.635+0000 7f12e867b640 1 Processor -- start 2024-04-25T01:22:53.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.635+0000 7f12e867b640 1 -- start start 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.636+0000 7f12e867b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e00cfaf0 0x7f12e00d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.636+0000 7f12e867b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12e00d1e90 con 0x7f12e00d4a10 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.636+0000 7f12e867b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12e00b54a0 con 0x7f12e015e4c0 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.636+0000 7f12e867b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12e00d2130 con 0x7f12e00cfaf0 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5c16640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f12e00d4a10 0x7f12e00b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36150/0 (socket says 172.21.15.77:36150) 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5c16640 1 -- 172.21.15.77:0/3313400788 learned_addr learned my addr 172.21.15.77:0/3313400788 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 957084581 0 0) 0x7f12e00d1e90 con 0x7f12e00d4a10 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12cc003520 con 0x7f12e00d4a10 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e6c18640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e00cfaf0 0x7f12e00d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2182899281 0 0) 0x7f12cc003520 con 0x7f12e00d4a10 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e00cfaf0 msgr2=0x7f12e00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:53.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e00cfaf0 0x7f12e00d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 >> v1:172.21.15.121:6789/0 conn(0x7f12e015e4c0 legacy=0x7f12e00d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e00b5780 con 0x7f12e00d4a10 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f12d4001670 con 0x7f12e00d4a10 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 >> v1:172.21.15.77:6789/0 conn(0x7f12e00d4a10 legacy=0x7f12e00b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12cc0039e0 0x7f12cc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e00cfaf0 0x7f12cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12cc004d40 0x7f12cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f12e00d2130 con 0x7f12cc004d40 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc00bba0 con 0x7f12cc0039e0 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12cc00bda0 con 0x7f12e00cfaf0 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e6c18640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12cc004d40 0x7f12cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e6417640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12cc0039e0 0x7f12cc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.637+0000 7f12e5c16640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e00cfaf0 0x7f12cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.638+0000 7f12e6c18640 1 -- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e00cfaf0 msgr2=0x7f12cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.638+0000 7f12e6c18640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e00cfaf0 0x7f12cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.638+0000 7f12e6c18640 1 -- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12cc0039e0 msgr2=0x7f12cc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.638+0000 7f12e6c18640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12cc0039e0 0x7f12cc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.638+0000 7f12e6c18640 1 -- 172.21.15.77:0/3313400788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12cc003520 con 0x7f12cc004d40 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.638+0000 7f12e5c16640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e00cfaf0 0x7f12cc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.638+0000 7f12e6c18640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12cc004d40 0x7f12cc0051c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f12d8007360 tx=0x7f12cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3cdb9177f548fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.638+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d8009be0 con 0x7f12cc004d40 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.638+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12d8036600 con 0x7f12cc004d40 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.638+0000 7f12e5415640 1 -- 172.21.15.77:0/3313400788 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d80369d0 con 0x7f12cc004d40 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.639+0000 7f12e867b640 1 -- 172.21.15.77:0/3313400788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12cc004d40 msgr2=0x7f12cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.639+0000 7f12e867b640 1 --2- 172.21.15.77:0/3313400788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12cc004d40 0x7f12cc0051c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f12d8007360 tx=0x7f12cc005830 comp rx=0 tx=0).stop 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.639+0000 7f12e6c18640 1 -- 172.21.15.77:0/3313400788 reap_dead start 2024-04-25T01:22:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.639+0000 7f12e867b640 1 -- 172.21.15.77:0/3313400788 shutdown_connections 2024-04-25T01:22:53.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.639+0000 7f12e867b640 1 -- 172.21.15.77:0/3313400788 >> 172.21.15.77:0/3313400788 conn(0x7f12e00d3ef0 msgr2=0x7f12e00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:53.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.639+0000 7f12e867b640 1 -- 172.21.15.77:0/3313400788 shutdown_connections 2024-04-25T01:22:53.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.639+0000 7f12e867b640 1 -- 172.21.15.77:0/3313400788 wait complete. 2024-04-25T01:22:53.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.640+0000 7f12e867b640 1 Processor -- start 2024-04-25T01:22:53.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.640+0000 7f12e867b640 1 -- start start 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e867b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12cc004d40 0x7f12e0152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e867b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12e01573e0 0x7f12e0159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e867b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12e01614a0 0x7f12e0159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e867b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f12e00b30a0 con 0x7f12e01614a0 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e867b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f12e00b2f20 con 0x7f12e01573e0 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e867b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12e015e8a0 con 0x7f12cc004d40 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e6c18640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12e01614a0 0x7f12e0159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e6c18640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12e01614a0 0x7f12e0159a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48630/0 (socket says 172.21.15.77:48630) 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e6c18640 1 -- 172.21.15.77:0/3053643933 learned_addr learned my addr 172.21.15.77:0/3053643933 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e6c18640 1 -- 172.21.15.77:0/3053643933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12cc004d40 msgr2=0x7f12e0152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e6417640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12cc004d40 0x7f12e0152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e5c16640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12e01573e0 0x7f12e0159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e6c18640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12cc004d40 0x7f12e0152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e6c18640 1 -- 172.21.15.77:0/3053643933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12e01573e0 msgr2=0x7f12e0159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e6c18640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12e01573e0 0x7f12e0159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e6c18640 1 -- 172.21.15.77:0/3053643933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e015f960 con 0x7f12e01614a0 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e6417640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12cc004d40 0x7f12e0152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e5c16640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12e01573e0 0x7f12e0159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.641+0000 7f12e6c18640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12e01614a0 0x7f12e0159a60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f12d802b070 tx=0x7f12d8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.642+0000 7f12d2ffd640 1 -- 172.21.15.77:0/3053643933 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d8045070 con 0x7f12e01614a0 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.642+0000 7f12d2ffd640 1 -- 172.21.15.77:0/3053643933 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12d8003990 con 0x7f12e01614a0 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.642+0000 7f12e867b640 1 -- 172.21.15.77:0/3053643933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e015fc40 con 0x7f12e01614a0 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.642+0000 7f12d2ffd640 1 -- 172.21.15.77:0/3053643933 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d8009be0 con 0x7f12e01614a0 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.642+0000 7f12e867b640 1 -- 172.21.15.77:0/3053643933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e0160200 con 0x7f12e01614a0 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.643+0000 7f12e867b640 1 -- 172.21.15.77:0/3053643933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12e0068fb0 con 0x7f12e01614a0 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.643+0000 7f12d2ffd640 1 -- 172.21.15.77:0/3053643933 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f12d802e830 con 0x7f12e01614a0 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.643+0000 7f12d2ffd640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f12c406be60 0x7f12c406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.643+0000 7f12d2ffd640 1 -- 172.21.15.77:0/3053643933 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f12d80ab390 con 0x7f12e01614a0 2024-04-25T01:22:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.643+0000 7f12e6417640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f12c406be60 0x7f12c406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.644+0000 7f12e6417640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f12c406be60 0x7f12c406e320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f12dc000c00 tx=0x7f12dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.648+0000 7f12d2ffd640 1 -- 172.21.15.77:0/3053643933 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f12d803d070 con 0x7f12e01614a0 2024-04-25T01:22:53.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.886+0000 7f12e867b640 1 -- 172.21.15.77:0/3053643933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f12e00b4720 con 0x7f12e01614a0 2024-04-25T01:22:53.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.888+0000 7f12d2ffd640 1 -- 172.21.15.77:0/3053643933 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7f12d8032030 con 0x7f12e01614a0 2024-04-25T01:22:53.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 -- 172.21.15.77:0/3053643933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f12c406be60 msgr2=0x7f12c406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f12c406be60 0x7f12c406e320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f12dc000c00 tx=0x7f12dc009040 comp rx=0 tx=0).stop 2024-04-25T01:22:53.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 -- 172.21.15.77:0/3053643933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12e01614a0 msgr2=0x7f12e0159a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12e01614a0 0x7f12e0159a60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f12d802b070 tx=0x7f12d8004ab0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 -- 172.21.15.77:0/3053643933 shutdown_connections 2024-04-25T01:22:53.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f12c406be60 0x7f12c406e320 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12e01614a0 0x7f12e0159a60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12e01573e0 0x7f12e0159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 --2- 172.21.15.77:0/3053643933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12cc004d40 0x7f12e0152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 -- 172.21.15.77:0/3053643933 >> 172.21.15.77:0/3053643933 conn(0x7f12e00d3ef0 msgr2=0x7f12e006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:53.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 -- 172.21.15.77:0/3053643933 shutdown_connections 2024-04-25T01:22:53.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.891+0000 7f12e867b640 1 -- 172.21.15.77:0/3053643933 wait complete. 2024-04-25T01:22:53.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-25T01:22:53.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.998+0000 7f8211c12640 1 Processor -- start 2024-04-25T01:22:53.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.998+0000 7f8211c12640 1 -- start start 2024-04-25T01:22:53.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.998+0000 7f8211c12640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c143940 0x7f820c06e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.998+0000 7f8211c12640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820c070e60 con 0x7f820c15f400 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.998+0000 7f8211c12640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820c071140 con 0x7f820c143d20 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.998+0000 7f8211c12640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f820c071420 con 0x7f820c143940 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f820c15f400 0x7f820c071c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36152/0 (socket says 172.21.15.77:36152) 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820bfff640 1 -- 172.21.15.77:0/2851749572 learned_addr learned my addr 172.21.15.77:0/2851749572 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820b7fe640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c143940 0x7f820c06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2477711513 0 0) 0x7f820c070e60 con 0x7f820c15f400 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81f4005730 con 0x7f820c15f400 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 22911998 0 0) 0x7f820c071140 con 0x7f820c143d20 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f820c070e60 con 0x7f820c143d20 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2017153035 0 0) 0x7f81f4005730 con 0x7f820c15f400 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c143940 msgr2=0x7f820c06e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:53.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c143940 0x7f820c06e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 >> v1:172.21.15.121:6789/0 conn(0x7f820c143d20 legacy=0x7f820c15ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c071700 con 0x7f820c15f400 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f82040019f0 con 0x7f820c15f400 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820b7fe640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c143940 0x7f820c06e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 >> v1:172.21.15.77:6789/0 conn(0x7f820c15f400 legacy=0x7f820c071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 0x7f81f4006180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f4006a10 0x7f81f4006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c143940 0x7f81f40073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f820c071420 con 0x7f81f4005d80 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f81f400dc80 con 0x7f81f4006a10 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f81f400de50 con 0x7f820c143940 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820b7fe640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 0x7f81f4006180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820affd640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f4006a10 0x7f81f4006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820bfff640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c143940 0x7f81f40073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820b7fe640 1 -- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c143940 msgr2=0x7f81f40073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820b7fe640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c143940 0x7f81f40073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820b7fe640 1 -- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f4006a10 msgr2=0x7f81f4006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:53.999+0000 7f820b7fe640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f4006a10 0x7f81f4006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.000+0000 7f820b7fe640 1 -- 172.21.15.77:0/2851749572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f4005730 con 0x7f81f4005d80 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.000+0000 7f820b7fe640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 0x7f81f4006180 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009c60 tx=0x7f81f4007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90251c32a867e92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.000+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc036070 con 0x7f81f4005d80 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.000+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81fc031400 con 0x7f81f4005d80 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.000+0000 7f820a7fc640 1 -- 172.21.15.77:0/2851749572 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc009860 con 0x7f81f4005d80 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.001+0000 7f8211c12640 1 -- 172.21.15.77:0/2851749572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 msgr2=0x7f81f4006180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.001+0000 7f8211c12640 1 --2- 172.21.15.77:0/2851749572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 0x7f81f4006180 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009c60 tx=0x7f81f4007a40 comp rx=0 tx=0).stop 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.001+0000 7f820bfff640 1 -- 172.21.15.77:0/2851749572 reap_dead start 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.001+0000 7f8211c12640 1 -- 172.21.15.77:0/2851749572 shutdown_connections 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.001+0000 7f8211c12640 1 -- 172.21.15.77:0/2851749572 >> 172.21.15.77:0/2851749572 conn(0x7f820c0b1820 msgr2=0x7f820c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.001+0000 7f8211c12640 1 -- 172.21.15.77:0/2851749572 shutdown_connections 2024-04-25T01:22:53.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.001+0000 7f8211c12640 1 -- 172.21.15.77:0/2851749572 wait complete. 2024-04-25T01:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.002+0000 7f8211c12640 1 Processor -- start 2024-04-25T01:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.002+0000 7f8211c12640 1 -- start start 2024-04-25T01:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f8211c12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 0x7f820c1429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f8211c12640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c0ae4b0 0x7f820c1f83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f8211c12640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f820c1f43f0 0x7f820c1f4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f8211c12640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f820c0726c0 con 0x7f81f4005d80 2024-04-25T01:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f8211c12640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f820c072540 con 0x7f820c1f43f0 2024-04-25T01:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f8211c12640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f820c072840 con 0x7f820c0ae4b0 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f820b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 0x7f820c1429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f820b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 0x7f820c1429a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48642/0 (socket says 172.21.15.77:48642) 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f820b7fe640 1 -- 172.21.15.77:0/1433390046 learned_addr learned my addr 172.21.15.77:0/1433390046 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f820b7fe640 1 -- 172.21.15.77:0/1433390046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c0ae4b0 msgr2=0x7f820c1f83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f820b7fe640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c0ae4b0 0x7f820c1f83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f820b7fe640 1 -- 172.21.15.77:0/1433390046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f820c1f43f0 msgr2=0x7f820c1f4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f820b7fe640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f820c1f43f0 0x7f820c1f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f820b7fe640 1 -- 172.21.15.77:0/1433390046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c0af370 con 0x7f81f4005d80 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f820b7fe640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 0x7f820c1429a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009d80 tx=0x7f81fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f81f3fff640 1 -- 172.21.15.77:0/1433390046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc040020 con 0x7f81f4005d80 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.003+0000 7f81f3fff640 1 -- 172.21.15.77:0/1433390046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81fc0093e0 con 0x7f81f4005d80 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.004+0000 7f81f3fff640 1 -- 172.21.15.77:0/1433390046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc036040 con 0x7f81f4005d80 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.004+0000 7f8211c12640 1 -- 172.21.15.77:0/1433390046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820c0aead0 con 0x7f81f4005d80 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.004+0000 7f8211c12640 1 -- 172.21.15.77:0/1433390046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f820c0af090 con 0x7f81f4005d80 2024-04-25T01:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.004+0000 7f8211c12640 1 -- 172.21.15.77:0/1433390046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f820c068fb0 con 0x7f81f4005d80 2024-04-25T01:22:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.005+0000 7f81f3fff640 1 -- 172.21.15.77:0/1433390046 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f81fc043020 con 0x7f81f4005d80 2024-04-25T01:22:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.005+0000 7f81f3fff640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f81f806be60 0x7f81f806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.005+0000 7f81f3fff640 1 -- 172.21.15.77:0/1433390046 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f81fc0ab230 con 0x7f81f4005d80 2024-04-25T01:22:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.005+0000 7f820affd640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f81f806be60 0x7f81f806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.006+0000 7f820affd640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f81f806be60 0x7f81f806e320 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8200007fd0 tx=0x7f8200009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:53.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.010+0000 7f81f3fff640 1 -- 172.21.15.77:0/1433390046 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f81fc04a030 con 0x7f81f4005d80 2024-04-25T01:22:54.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.251+0000 7f8211c12640 1 -- 172.21.15.77:0/1433390046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f820c143160 con 0x7f81f4005d80 2024-04-25T01:22:54.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.253+0000 7f81f3fff640 1 -- 172.21.15.77:0/1433390046 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f81fc079da0 con 0x7f81f4005d80 2024-04-25T01:22:54.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 -- 172.21.15.77:0/1433390046 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f81f806be60 msgr2=0x7f81f806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f81f806be60 0x7f81f806e320 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8200007fd0 tx=0x7f8200009040 comp rx=0 tx=0).stop 2024-04-25T01:22:54.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 -- 172.21.15.77:0/1433390046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 msgr2=0x7f820c1429a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 0x7f820c1429a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009d80 tx=0x7f81fc004730 comp rx=0 tx=0).stop 2024-04-25T01:22:54.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 -- 172.21.15.77:0/1433390046 shutdown_connections 2024-04-25T01:22:54.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f81f806be60 0x7f81f806e320 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f820c1f43f0 0x7f820c1f4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f820c0ae4b0 0x7f820c1f83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 --2- 172.21.15.77:0/1433390046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f4005d80 0x7f820c1429a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 -- 172.21.15.77:0/1433390046 >> 172.21.15.77:0/1433390046 conn(0x7f820c0b1820 msgr2=0x7f820c0af690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 -- 172.21.15.77:0/1433390046 shutdown_connections 2024-04-25T01:22:54.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.255+0000 7f8211c12640 1 -- 172.21.15.77:0/1433390046 wait complete. 2024-04-25T01:22:54.247 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:54.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.362+0000 7f7fd56f5640 1 Processor -- start 2024-04-25T01:22:54.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.362+0000 7f7fd56f5640 1 -- start start 2024-04-25T01:22:54.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.362+0000 7f7fd56f5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fd015e4c0 0x7f7fd00d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.362+0000 7f7fd56f5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7fd00d2d40 0x7f7fd00cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.362+0000 7f7fd56f5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fd00d1e90 con 0x7f7fd00d4a10 2024-04-25T01:22:54.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.362+0000 7f7fd56f5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f7fd00b54a0 con 0x7f7fd015e4c0 2024-04-25T01:22:54.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.362+0000 7f7fd56f5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7fd00d2130 con 0x7f7fd00d2d40 2024-04-25T01:22:54.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.363+0000 7f7fceffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fd015e4c0 0x7f7fd00d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.363+0000 7f7fceffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fd015e4c0 0x7f7fd00d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48650/0 (socket says 172.21.15.77:48650) 2024-04-25T01:22:54.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.363+0000 7f7fceffd640 1 -- 172.21.15.77:0/3752599472 learned_addr learned my addr 172.21.15.77:0/3752599472 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:54.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.363+0000 7f7fceffd640 1 -- 172.21.15.77:0/3752599472 >> v2:172.21.15.121:3300/0 conn(0x7f7fd00d2d40 msgr2=0x7f7fd00cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:54.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.363+0000 7f7fceffd640 1 --2- 172.21.15.77:0/3752599472 >> v2:172.21.15.121:3300/0 conn(0x7f7fd00d2d40 0x7f7fd00cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.363+0000 7f7fceffd640 1 -- 172.21.15.77:0/3752599472 >> v1:172.21.15.121:6789/0 conn(0x7f7fd00d4a10 legacy=0x7f7fd00b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:54.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.363+0000 7f7fceffd640 1 -- 172.21.15.77:0/3752599472 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd00b5780 con 0x7f7fd015e4c0 2024-04-25T01:22:54.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fceffd640 1 --2- 172.21.15.77:0/3752599472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd015e4c0 0x7f7fd00d2800 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7fd008d2e0 tx=0x7f7fc002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c27780fd5d94328 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fcdffb640 1 -- 172.21.15.77:0/3752599472 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0037070 con 0x7f7fd015e4c0 2024-04-25T01:22:54.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fcdffb640 1 -- 172.21.15.77:0/3752599472 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fc0032440 con 0x7f7fd015e4c0 2024-04-25T01:22:54.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fcdffb640 1 -- 172.21.15.77:0/3752599472 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0032810 con 0x7f7fd015e4c0 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3752599472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd015e4c0 msgr2=0x7f7fd00d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fd56f5640 1 --2- 172.21.15.77:0/3752599472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd015e4c0 0x7f7fd00d2800 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7fd008d2e0 tx=0x7f7fc002a1a0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3752599472 shutdown_connections 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fd56f5640 1 --2- 172.21.15.77:0/3752599472 >> v2:172.21.15.121:3300/0 conn(0x7f7fd00d2d40 0x7f7fd00cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fd56f5640 1 --2- 172.21.15.77:0/3752599472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd015e4c0 0x7f7fd00d2800 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3752599472 >> 172.21.15.77:0/3752599472 conn(0x7f7fd00d3ef0 msgr2=0x7f7fd00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3752599472 shutdown_connections 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.364+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3752599472 wait complete. 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.365+0000 7f7fd56f5640 1 Processor -- start 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.365+0000 7f7fd56f5640 1 -- start start 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.366+0000 7f7fd56f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd00d2d40 0x7f7fd00ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.366+0000 7f7fd56f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd00d4a10 0x7f7fd00cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.366+0000 7f7fd56f5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd015e4c0 0x7f7fd01523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.366+0000 7f7fd56f5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd015ea20 con 0x7f7fd00d4a10 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.366+0000 7f7fd56f5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd015e8a0 con 0x7f7fd015e4c0 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.366+0000 7f7fd56f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7fd00b45f0 con 0x7f7fd00d2d40 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.366+0000 7f7fce7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd00d4a10 0x7f7fd00cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.366+0000 7f7fce7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd00d4a10 0x7f7fd00cafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48656/0 (socket says 172.21.15.77:48656) 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.366+0000 7f7fce7fc640 1 -- 172.21.15.77:0/3812624724 learned_addr learned my addr 172.21.15.77:0/3812624724 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.366+0000 7f7fcf7fe640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd015e4c0 0x7f7fd01523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.366+0000 7f7fceffd640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd00d2d40 0x7f7fd00ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.367+0000 7f7fce7fc640 1 -- 172.21.15.77:0/3812624724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd00d2d40 msgr2=0x7f7fd00ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.367+0000 7f7fce7fc640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd00d2d40 0x7f7fd00ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.367+0000 7f7fce7fc640 1 -- 172.21.15.77:0/3812624724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd015e4c0 msgr2=0x7f7fd01523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.367+0000 7f7fce7fc640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd015e4c0 0x7f7fd01523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.367+0000 7f7fce7fc640 1 -- 172.21.15.77:0/3812624724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd00c9410 con 0x7f7fd00d4a10 2024-04-25T01:22:54.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.367+0000 7f7fce7fc640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd00d4a10 0x7f7fd00cafc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400c430 tx=0x7f7fc400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.367+0000 7f7fb77fe640 1 -- 172.21.15.77:0/3812624724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4017070 con 0x7f7fd00d4a10 2024-04-25T01:22:54.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.367+0000 7f7fb77fe640 1 -- 172.21.15.77:0/3812624724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fc40035f0 con 0x7f7fd00d4a10 2024-04-25T01:22:54.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.367+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3812624724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd00c96f0 con 0x7f7fd00d4a10 2024-04-25T01:22:54.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.367+0000 7f7fb77fe640 1 -- 172.21.15.77:0/3812624724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc40039e0 con 0x7f7fd00d4a10 2024-04-25T01:22:54.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.367+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3812624724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd00c9cb0 con 0x7f7fd00d4a10 2024-04-25T01:22:54.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.368+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3812624724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd0068fb0 con 0x7f7fd00d4a10 2024-04-25T01:22:54.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.368+0000 7f7fb77fe640 1 -- 172.21.15.77:0/3812624724 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7fc401a070 con 0x7f7fd00d4a10 2024-04-25T01:22:54.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.369+0000 7f7fb77fe640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fbc06be60 0x7f7fbc06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.369+0000 7f7fb77fe640 1 -- 172.21.15.77:0/3812624724 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f7fc4089f70 con 0x7f7fd00d4a10 2024-04-25T01:22:54.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.369+0000 7f7fceffd640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fbc06be60 0x7f7fbc06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.369+0000 7f7fceffd640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fbc06be60 0x7f7fbc06e320 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0034000 tx=0x7f7fc002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.373+0000 7f7fb77fe640 1 -- 172.21.15.77:0/3812624724 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7fc4058490 con 0x7f7fd00d4a10 2024-04-25T01:22:54.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.547+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3812624724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7fd00b4080 con 0x7f7fd00d4a10 2024-04-25T01:22:54.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.548+0000 7f7fb77fe640 1 -- 172.21.15.77:0/3812624724 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+13164 (secure 0 0 0) 0x7f7fc405c790 con 0x7f7fd00d4a10 2024-04-25T01:22:54.530 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:54.530 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":34,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:51.701790+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3812624724 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fbc06be60 msgr2=0x7f7fbc06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fbc06be60 0x7f7fbc06e320 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0034000 tx=0x7f7fc002e000 comp rx=0 tx=0).stop 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3812624724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd00d4a10 msgr2=0x7f7fd00cafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd00d4a10 0x7f7fd00cafc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400c430 tx=0x7f7fc400c8f0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3812624724 shutdown_connections 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fbc06be60 0x7f7fbc06e320 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd00d4a10 0x7f7fd00cafc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd015e4c0 0x7f7fd01523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 --2- 172.21.15.77:0/3812624724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd00d2d40 0x7f7fd00ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3812624724 >> 172.21.15.77:0/3812624724 conn(0x7f7fd00d3ef0 msgr2=0x7f7fd00d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3812624724 shutdown_connections 2024-04-25T01:22:54.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.550+0000 7f7fd56f5640 1 -- 172.21.15.77:0/3812624724 wait complete. 2024-04-25T01:22:54.542 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:22:54.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:54.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.661+0000 7f2c6aeb3640 1 Processor -- start 2024-04-25T01:22:54.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.661+0000 7f2c6aeb3640 1 -- start start 2024-04-25T01:22:54.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c6aeb3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2c640cf4d0 0x7f2c641600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c6aeb3640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6406d110 con 0x7f2c6415fc40 2024-04-25T01:22:54.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c6aeb3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6406d3f0 con 0x7f2c640722e0 2024-04-25T01:22:54.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c6aeb3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2c64160aa0 con 0x7f2c640cf4d0 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c63fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2c6415fc40 0x7f2c640cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36160/0 (socket says 172.21.15.77:36160) 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c63fff640 1 -- 172.21.15.77:0/385745426 learned_addr learned my addr 172.21.15.77:0/385745426 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c69450640 1 --2- 172.21.15.77:0/385745426 >> v2:172.21.15.121:3300/0 conn(0x7f2c640cf4d0 0x7f2c641600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 69441012 0 0) 0x7f2c6406d110 con 0x7f2c6415fc40 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c54003520 con 0x7f2c6415fc40 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3923002521 0 0) 0x7f2c6406d3f0 con 0x7f2c640722e0 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c6406d110 con 0x7f2c640722e0 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1674645088 0 0) 0x7f2c54003520 con 0x7f2c6415fc40 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 >> v2:172.21.15.121:3300/0 conn(0x7f2c640cf4d0 msgr2=0x7f2c641600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 --2- 172.21.15.77:0/385745426 >> v2:172.21.15.121:3300/0 conn(0x7f2c640cf4d0 0x7f2c641600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 >> v1:172.21.15.121:6789/0 conn(0x7f2c640722e0 legacy=0x7f2c6406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6406d6d0 con 0x7f2c6415fc40 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f2c50001670 con 0x7f2c6415fc40 2024-04-25T01:22:54.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c69450640 1 --2- 172.21.15.77:0/385745426 >> v2:172.21.15.121:3300/0 conn(0x7f2c640cf4d0 0x7f2c641600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 >> v1:172.21.15.77:6789/0 conn(0x7f2c6415fc40 legacy=0x7f2c640cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 --2- 172.21.15.77:0/385745426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 0x7f2c54003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 --2- 172.21.15.77:0/385745426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c54004800 0x7f2c54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 --2- 172.21.15.77:0/385745426 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c540051c0 0x7f2c5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c64160aa0 con 0x7f2c54003b40 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5400c0f0 con 0x7f2c54004800 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2c5400c2f0 con 0x7f2c540051c0 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c68c4f640 1 --2- 172.21.15.77:0/385745426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 0x7f2c54003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.662+0000 7f2c63fff640 1 --2- 172.21.15.77:0/385745426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c54004800 0x7f2c54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.663+0000 7f2c69450640 1 --2- 172.21.15.77:0/385745426 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c540051c0 0x7f2c5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.663+0000 7f2c68c4f640 1 -- 172.21.15.77:0/385745426 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c540051c0 msgr2=0x7f2c5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.663+0000 7f2c68c4f640 1 --2- 172.21.15.77:0/385745426 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c540051c0 0x7f2c5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.663+0000 7f2c68c4f640 1 -- 172.21.15.77:0/385745426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c54004800 msgr2=0x7f2c54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.663+0000 7f2c68c4f640 1 --2- 172.21.15.77:0/385745426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c54004800 0x7f2c54004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.663+0000 7f2c69450640 1 -- 172.21.15.77:0/385745426 reap_dead start 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.663+0000 7f2c68c4f640 1 -- 172.21.15.77:0/385745426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c54003520 con 0x7f2c54003b40 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.663+0000 7f2c68c4f640 1 --2- 172.21.15.77:0/385745426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 0x7f2c54003f40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2c580018d0 tx=0x7f2c54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f01f31c55fed7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.663+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c580014f0 con 0x7f2c54003b40 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.663+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c580052d0 con 0x7f2c54003b40 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.663+0000 7f2c637fe640 1 -- 172.21.15.77:0/385745426 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5802ac40 con 0x7f2c54003b40 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.664+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/385745426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 msgr2=0x7f2c54003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.664+0000 7f2c6aeb3640 1 --2- 172.21.15.77:0/385745426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 0x7f2c54003f40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2c580018d0 tx=0x7f2c54005830 comp rx=0 tx=0).stop 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.664+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/385745426 shutdown_connections 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.664+0000 7f2c6aeb3640 1 --2- 172.21.15.77:0/385745426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 0x7f2c54003f40 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.664+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/385745426 >> 172.21.15.77:0/385745426 conn(0x7f2c6406b720 msgr2=0x7f2c640b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.664+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/385745426 shutdown_connections 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.664+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/385745426 wait complete. 2024-04-25T01:22:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.665+0000 7f2c6aeb3640 1 Processor -- start 2024-04-25T01:22:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.665+0000 7f2c6aeb3640 1 -- start start 2024-04-25T01:22:54.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c6aeb3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 0x7f2c641431e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c6aeb3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c64142c90 0x7f2c641593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c6aeb3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c641e6cd0 0x7f2c641e7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c6aeb3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c64160430 con 0x7f2c54003b40 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c6aeb3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2c641602b0 con 0x7f2c64142c90 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c6aeb3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2c641605b0 con 0x7f2c641e6cd0 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c68c4f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 0x7f2c641431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c68c4f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 0x7f2c641431e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48670/0 (socket says 172.21.15.77:48670) 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c68c4f640 1 -- 172.21.15.77:0/2208528606 learned_addr learned my addr 172.21.15.77:0/2208528606 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c63fff640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c64142c90 0x7f2c641593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c69450640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c641e6cd0 0x7f2c641e7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c68c4f640 1 -- 172.21.15.77:0/2208528606 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c641e6cd0 msgr2=0x7f2c641e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c68c4f640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c641e6cd0 0x7f2c641e7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c68c4f640 1 -- 172.21.15.77:0/2208528606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c64142c90 msgr2=0x7f2c641593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c68c4f640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c64142c90 0x7f2c641593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c68c4f640 1 -- 172.21.15.77:0/2208528606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c64159b70 con 0x7f2c54003b40 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.666+0000 7f2c69450640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c641e6cd0 0x7f2c641e7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:54.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.667+0000 7f2c68c4f640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 0x7f2c641431e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2c58000c00 tx=0x7f2c580012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.667+0000 7f2c617fa640 1 -- 172.21.15.77:0/2208528606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58037040 con 0x7f2c54003b40 2024-04-25T01:22:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.667+0000 7f2c617fa640 1 -- 172.21.15.77:0/2208528606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c58006e50 con 0x7f2c54003b40 2024-04-25T01:22:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.667+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/2208528606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c641598f0 con 0x7f2c54003b40 2024-04-25T01:22:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.667+0000 7f2c617fa640 1 -- 172.21.15.77:0/2208528606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c580014f0 con 0x7f2c54003b40 2024-04-25T01:22:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.667+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/2208528606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c641e61f0 con 0x7f2c54003b40 2024-04-25T01:22:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.668+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/2208528606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c64068fb0 con 0x7f2c54003b40 2024-04-25T01:22:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.668+0000 7f2c617fa640 1 -- 172.21.15.77:0/2208528606 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2c58044020 con 0x7f2c54003b40 2024-04-25T01:22:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.668+0000 7f2c617fa640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c4806be30 0x7f2c4806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.669+0000 7f2c617fa640 1 -- 172.21.15.77:0/2208528606 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f2c580aa790 con 0x7f2c54003b40 2024-04-25T01:22:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.669+0000 7f2c63fff640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c4806be30 0x7f2c4806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.669+0000 7f2c63fff640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c4806be30 0x7f2c4806e2f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2c50001200 tx=0x7f2c50001e40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.673+0000 7f2c617fa640 1 -- 172.21.15.77:0/2208528606 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2c5804b020 con 0x7f2c54003b40 2024-04-25T01:22:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.848+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/2208528606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2c64072c90 con 0x7f2c54003b40 2024-04-25T01:22:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.849+0000 7f2c617fa640 1 -- 172.21.15.77:0/2208528606 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7f2c641598f0 con 0x7f2c54003b40 2024-04-25T01:22:54.830 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:22:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.850+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/2208528606 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c4806be30 msgr2=0x7f2c4806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.850+0000 7f2c6aeb3640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c4806be30 0x7f2c4806e2f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2c50001200 tx=0x7f2c50001e40 comp rx=0 tx=0).stop 2024-04-25T01:22:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.850+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/2208528606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 msgr2=0x7f2c641431e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.850+0000 7f2c6aeb3640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 0x7f2c641431e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2c58000c00 tx=0x7f2c580012d0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.851+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/2208528606 shutdown_connections 2024-04-25T01:22:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.851+0000 7f2c6aeb3640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2c4806be30 0x7f2c4806e2f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.851+0000 7f2c6aeb3640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c641e6cd0 0x7f2c641e7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.851+0000 7f2c6aeb3640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c64142c90 0x7f2c641593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.851+0000 7f2c6aeb3640 1 --2- 172.21.15.77:0/2208528606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c54003b40 0x7f2c641431e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.851+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/2208528606 >> 172.21.15.77:0/2208528606 conn(0x7f2c6406b720 msgr2=0x7f2c640b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.851+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/2208528606 shutdown_connections 2024-04-25T01:22:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.851+0000 7f2c6aeb3640 1 -- 172.21.15.77:0/2208528606 wait complete. 2024-04-25T01:22:54.843 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T01:22:54.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.958+0000 7fe7c259b640 1 Processor -- start 2024-04-25T01:22:54.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.958+0000 7fe7c259b640 1 -- start start 2024-04-25T01:22:54.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.959+0000 7fe7c259b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc15de10 0x7fe7bc161c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.959+0000 7fe7c259b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7bc0b3a10 con 0x7fe7bc15e1f0 2024-04-25T01:22:54.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.959+0000 7fe7c259b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7bc0b3cf0 con 0x7fe7bc0b4db0 2024-04-25T01:22:54.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.959+0000 7fe7c259b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe7bc0b3fd0 con 0x7fe7bc15de10 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.959+0000 7fe7c0b38640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe7bc15e1f0 0x7fe7bc0719c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36170/0 (socket says 172.21.15.77:36170) 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.959+0000 7fe7c0b38640 1 -- 172.21.15.77:0/10523678 learned_addr learned my addr 172.21.15.77:0/10523678 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.959+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2131339930 0 0) 0x7fe7bc0b3a10 con 0x7fe7bc15e1f0 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.959+0000 7fe7bbfff640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc15de10 0x7fe7bc161c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.959+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7a4003520 con 0x7fe7bc15e1f0 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4127215292 0 0) 0x7fe7a4003520 con 0x7fe7bc15e1f0 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc15de10 msgr2=0x7fe7bc161c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc15de10 0x7fe7bc161c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 >> v1:172.21.15.121:6789/0 conn(0x7fe7bc0b4db0 legacy=0x7fe7bc162190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7bc0b4660 con 0x7fe7bc15e1f0 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fe7b40019f0 con 0x7fe7bc15e1f0 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 >> v1:172.21.15.77:6789/0 conn(0x7fe7bc15e1f0 legacy=0x7fe7bc0719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7a40039e0 0x7fe7a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a4004800 0x7fe7a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc15de10 0x7fe7a4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc0b3fd0 con 0x7fe7a40039e0 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a400baf0 con 0x7fe7a4004800 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a400bcf0 con 0x7fe7bc15de10 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7bbfff640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7a40039e0 0x7fe7a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7bb7fe640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a4004800 0x7fe7a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7c0b38640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc15de10 0x7fe7a4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7bbfff640 1 -- 172.21.15.77:0/10523678 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc15de10 msgr2=0x7fe7a4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7bbfff640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc15de10 0x7fe7a4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7bbfff640 1 -- 172.21.15.77:0/10523678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a4004800 msgr2=0x7fe7a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7bbfff640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a4004800 0x7fe7a4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.960+0000 7fe7bbfff640 1 -- 172.21.15.77:0/10523678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a4003520 con 0x7fe7a40039e0 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.961+0000 7fe7bbfff640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7a40039e0 0x7fe7a4003dc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac009ac0 tx=0x7fe7a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8db86f4a587e12a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.961+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac036070 con 0x7fe7a40039e0 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.961+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7ac031400 con 0x7fe7a40039e0 2024-04-25T01:22:54.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.961+0000 7fe7baffd640 1 -- 172.21.15.77:0/10523678 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac009690 con 0x7fe7a40039e0 2024-04-25T01:22:54.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.961+0000 7fe7c259b640 1 -- 172.21.15.77:0/10523678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7a40039e0 msgr2=0x7fe7a4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.961+0000 7fe7c259b640 1 --2- 172.21.15.77:0/10523678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7a40039e0 0x7fe7a4003dc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac009ac0 tx=0x7fe7a4005830 comp rx=0 tx=0).stop 2024-04-25T01:22:54.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.961+0000 7fe7c0b38640 1 -- 172.21.15.77:0/10523678 reap_dead start 2024-04-25T01:22:54.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.961+0000 7fe7c259b640 1 -- 172.21.15.77:0/10523678 shutdown_connections 2024-04-25T01:22:54.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.961+0000 7fe7c259b640 1 -- 172.21.15.77:0/10523678 >> 172.21.15.77:0/10523678 conn(0x7fe7bc0b1820 msgr2=0x7fe7bc0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.962+0000 7fe7c259b640 1 -- 172.21.15.77:0/10523678 shutdown_connections 2024-04-25T01:22:54.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.962+0000 7fe7c259b640 1 -- 172.21.15.77:0/10523678 wait complete. 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.963+0000 7fe7c259b640 1 Processor -- start 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.963+0000 7fe7c259b640 1 -- start start 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7c259b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc1464c0 0x7fe7bc160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7c259b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc1468a0 0x7fe7bc1e92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7c259b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7bc1e9800 0x7fe7bc0cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7c259b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc161c50 con 0x7fe7bc1464c0 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7c259b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc162ab0 con 0x7fe7bc1e9800 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7c259b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe7bc161dd0 con 0x7fe7bc1468a0 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7bbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc1464c0 0x7fe7bc160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7bbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc1464c0 0x7fe7bc160f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48690/0 (socket says 172.21.15.77:48690) 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7bbfff640 1 -- 172.21.15.77:0/2755216357 learned_addr learned my addr 172.21.15.77:0/2755216357 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7bbfff640 1 -- 172.21.15.77:0/2755216357 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc1468a0 msgr2=0x7fe7bc1e92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7bbfff640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc1468a0 0x7fe7bc1e92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7bbfff640 1 -- 172.21.15.77:0/2755216357 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7bc1e9800 msgr2=0x7fe7bc0cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7bbfff640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7bc1e9800 0x7fe7bc0cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7bbfff640 1 -- 172.21.15.77:0/2755216357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7bc15fec0 con 0x7fe7bc1464c0 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7bbfff640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc1464c0 0x7fe7bc160f60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac009be0 tx=0x7fe7ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7b8ff9640 1 -- 172.21.15.77:0/2755216357 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac036050 con 0x7fe7bc1464c0 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7b8ff9640 1 -- 172.21.15.77:0/2755216357 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7ac042070 con 0x7fe7bc1464c0 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7c259b640 1 -- 172.21.15.77:0/2755216357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7bc1601a0 con 0x7fe7bc1464c0 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7b8ff9640 1 -- 172.21.15.77:0/2755216357 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac003c40 con 0x7fe7bc1464c0 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.964+0000 7fe7c259b640 1 -- 172.21.15.77:0/2755216357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7bc160760 con 0x7fe7bc1464c0 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.965+0000 7fe7c259b640 1 -- 172.21.15.77:0/2755216357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7bc068fb0 con 0x7fe7bc1464c0 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.966+0000 7fe7b8ff9640 1 -- 172.21.15.77:0/2755216357 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe7ac03d450 con 0x7fe7bc1464c0 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.966+0000 7fe7b8ff9640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7a806be60 0x7fe7a806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.966+0000 7fe7b8ff9640 1 -- 172.21.15.77:0/2755216357 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fe7ac0aadb0 con 0x7fe7bc1464c0 2024-04-25T01:22:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.966+0000 7fe7bb7fe640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7a806be60 0x7fe7a806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.966+0000 7fe7bb7fe640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7a806be60 0x7fe7a806e320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0007fd0 tx=0x7fe7b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:54.970+0000 7fe7b8ff9640 1 -- 172.21.15.77:0/2755216357 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe7ac04b020 con 0x7fe7bc1464c0 2024-04-25T01:22:55.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.143+0000 7fe7c259b640 1 -- 172.21.15.77:0/2755216357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe7bc070eb0 con 0x7fe7bc1464c0 2024-04-25T01:22:55.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.143+0000 7fe7b8ff9640 1 -- 172.21.15.77:0/2755216357 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) ==== 84+0+65 (secure 0 0 0) 0x7fe7ac079920 con 0x7fe7bc1464c0 2024-04-25T01:22:55.124 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:55.124 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-04-25T01:22:55.125 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:22:55.125 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-04-25T01:22:55.125 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-04-25T01:22:55.125 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:22:55.125 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-04-25T01:22:55.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 -- 172.21.15.77:0/2755216357 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7a806be60 msgr2=0x7fe7a806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7a806be60 0x7fe7a806e320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0007fd0 tx=0x7fe7b0009040 comp rx=0 tx=0).stop 2024-04-25T01:22:55.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 -- 172.21.15.77:0/2755216357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc1464c0 msgr2=0x7fe7bc160f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc1464c0 0x7fe7bc160f60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac009be0 tx=0x7fe7ac004730 comp rx=0 tx=0).stop 2024-04-25T01:22:55.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 -- 172.21.15.77:0/2755216357 shutdown_connections 2024-04-25T01:22:55.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7a806be60 0x7fe7a806e320 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7bc1e9800 0x7fe7bc0cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7bc1468a0 0x7fe7bc1e92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 --2- 172.21.15.77:0/2755216357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc1464c0 0x7fe7bc160f60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 -- 172.21.15.77:0/2755216357 >> 172.21.15.77:0/2755216357 conn(0x7fe7bc0b1820 msgr2=0x7fe7bc0b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:55.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 -- 172.21.15.77:0/2755216357 shutdown_connections 2024-04-25T01:22:55.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.145+0000 7fe7c259b640 1 -- 172.21.15.77:0/2755216357 wait complete. 2024-04-25T01:22:55.137 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T01:22:55.137 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T01:22:55.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.255+0000 7f87be516640 1 Processor -- start 2024-04-25T01:22:55.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.255+0000 7f87be516640 1 -- start start 2024-04-25T01:22:55.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87be516640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f87b80c8e30 0x7f87b80c9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87be516640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b80ce090 0x7f87b80ca130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87be516640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87b8147b00 con 0x7f87b80ca670 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87be516640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f87b8071e10 con 0x7f87b80c8e30 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87be516640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f87b8071800 con 0x7f87b80ce090 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87bcab3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f87b80ca670 0x7f87b80722a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36172/0 (socket says 172.21.15.77:36172) 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87bcab3640 1 -- 172.21.15.77:0/3757711333 learned_addr learned my addr 172.21.15.77:0/3757711333 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87b77fe640 1 --2- 172.21.15.77:0/3757711333 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b80ce090 0x7f87b80ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87b7fff640 1 --2- 172.21.15.77:0/3757711333 >> v2:172.21.15.121:3300/0 conn(0x7f87b80c8e30 0x7f87b80c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87b6ffd640 1 -- 172.21.15.77:0/3757711333 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1739983262 0 0) 0x7f87b8147b00 con 0x7f87b80ca670 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87b6ffd640 1 -- 172.21.15.77:0/3757711333 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87a0003520 con 0x7f87b80ca670 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87b6ffd640 1 -- 172.21.15.77:0/3757711333 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3309133107 0 0) 0x7f87a0003520 con 0x7f87b80ca670 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87b77fe640 1 -- 172.21.15.77:0/3757711333 >> v2:172.21.15.121:3300/0 conn(0x7f87b80c8e30 msgr2=0x7f87b80c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:55.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87b77fe640 1 --2- 172.21.15.77:0/3757711333 >> v2:172.21.15.121:3300/0 conn(0x7f87b80c8e30 0x7f87b80c9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87b77fe640 1 -- 172.21.15.77:0/3757711333 >> v1:172.21.15.77:6789/0 conn(0x7f87b80ca670 legacy=0x7f87b80722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.256+0000 7f87b77fe640 1 -- 172.21.15.77:0/3757711333 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b80c9750 con 0x7f87b80ce090 2024-04-25T01:22:55.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.257+0000 7f87b77fe640 1 --2- 172.21.15.77:0/3757711333 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b80ce090 0x7f87b80ca130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0097a0 tx=0x7f87ac02a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b711099ea311c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:55.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.257+0000 7f87b6ffd640 1 -- 172.21.15.77:0/3757711333 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac037070 con 0x7f87b80ce090 2024-04-25T01:22:55.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.257+0000 7f87b6ffd640 1 -- 172.21.15.77:0/3757711333 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87ac02aba0 con 0x7f87b80ce090 2024-04-25T01:22:55.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.257+0000 7f87b6ffd640 1 -- 172.21.15.77:0/3757711333 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b00019f0 con 0x7f87b80ce090 2024-04-25T01:22:55.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.258+0000 7f87be516640 1 -- 172.21.15.77:0/3757711333 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b80ce090 msgr2=0x7f87b80ca130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.258+0000 7f87be516640 1 --2- 172.21.15.77:0/3757711333 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b80ce090 0x7f87b80ca130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0097a0 tx=0x7f87ac02a110 comp rx=0 tx=0).stop 2024-04-25T01:22:55.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.258+0000 7f87be516640 1 -- 172.21.15.77:0/3757711333 shutdown_connections 2024-04-25T01:22:55.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.258+0000 7f87be516640 1 --2- 172.21.15.77:0/3757711333 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b80ce090 0x7f87b80ca130 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.258+0000 7f87be516640 1 --2- 172.21.15.77:0/3757711333 >> v2:172.21.15.121:3300/0 conn(0x7f87b80c8e30 0x7f87b80c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.258+0000 7f87be516640 1 -- 172.21.15.77:0/3757711333 >> 172.21.15.77:0/3757711333 conn(0x7f87b80cfae0 msgr2=0x7f87b80cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:55.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.258+0000 7f87be516640 1 -- 172.21.15.77:0/3757711333 shutdown_connections 2024-04-25T01:22:55.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.258+0000 7f87be516640 1 -- 172.21.15.77:0/3757711333 wait complete. 2024-04-25T01:22:55.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.259+0000 7f87be516640 1 Processor -- start 2024-04-25T01:22:55.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.259+0000 7f87be516640 1 -- start start 2024-04-25T01:22:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87be516640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b80c8e30 0x7f87b80cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87be516640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b80ca670 0x7f87b81f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87be516640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b80ce090 0x7f87b80b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87be516640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f87b80cf080 con 0x7f87b80c8e30 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87be516640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f87b80ceda0 con 0x7f87b80ce090 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87be516640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f87b8071aa0 con 0x7f87b80ca670 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87b7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b80c8e30 0x7f87b80cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87b7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b80c8e30 0x7f87b80cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48700/0 (socket says 172.21.15.77:48700) 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87b7fff640 1 -- 172.21.15.77:0/30859914 learned_addr learned my addr 172.21.15.77:0/30859914 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87bcab3640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b80ce090 0x7f87b80b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87b77fe640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b80ca670 0x7f87b81f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87b7fff640 1 -- 172.21.15.77:0/30859914 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b80ca670 msgr2=0x7f87b81f3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87b7fff640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b80ca670 0x7f87b81f3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87b7fff640 1 -- 172.21.15.77:0/30859914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b80ce090 msgr2=0x7f87b80b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87b7fff640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b80ce090 0x7f87b80b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87b7fff640 1 -- 172.21.15.77:0/30859914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b81f4970 con 0x7f87b80c8e30 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87bcab3640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b80ce090 0x7f87b80b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87b7fff640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b80c8e30 0x7f87b80cc5d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f87a800e8a0 tx=0x7f87a800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.260+0000 7f87b4ff9640 1 -- 172.21.15.77:0/30859914 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8017070 con 0x7f87b80c8e30 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.261+0000 7f87b4ff9640 1 -- 172.21.15.77:0/30859914 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a800c8e0 con 0x7f87b80c8e30 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.261+0000 7f87be516640 1 -- 172.21.15.77:0/30859914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87b81f4c50 con 0x7f87b80c8e30 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.261+0000 7f87b4ff9640 1 -- 172.21.15.77:0/30859914 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a800ccd0 con 0x7f87b80c8e30 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.261+0000 7f87be516640 1 -- 172.21.15.77:0/30859914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87b806cdd0 con 0x7f87b80c8e30 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.261+0000 7f87be516640 1 -- 172.21.15.77:0/30859914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87b8068fb0 con 0x7f87b80c8e30 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.262+0000 7f87b4ff9640 1 -- 172.21.15.77:0/30859914 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f87a801a070 con 0x7f87b80c8e30 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.262+0000 7f87b4ff9640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87a406be30 0x7f87a406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.262+0000 7f87b4ff9640 1 -- 172.21.15.77:0/30859914 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f87a808aed0 con 0x7f87b80c8e30 2024-04-25T01:22:55.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.262+0000 7f87b77fe640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87a406be30 0x7f87a406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.263+0000 7f87b77fe640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87a406be30 0x7f87a406e2f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f87ac034000 tx=0x7f87ac02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:55.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.267+0000 7f87b4ff9640 1 -- 172.21.15.77:0/30859914 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f87a8059a40 con 0x7f87b80c8e30 2024-04-25T01:22:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.438+0000 7f87be516640 1 -- 172.21.15.77:0/30859914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f87b80ce970 con 0x7f87b80c8e30 2024-04-25T01:22:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.705+0000 7f87b4ff9640 1 -- 172.21.15.77:0/30859914 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) ==== 138+0+0 (secure 0 0 0) 0x7f87a8059340 con 0x7f87b80c8e30 2024-04-25T01:22:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-04-25T01:22:55.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.707+0000 7f87be516640 1 -- 172.21.15.77:0/30859914 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87a406be30 msgr2=0x7f87a406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.707+0000 7f87be516640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87a406be30 0x7f87a406e2f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f87ac034000 tx=0x7f87ac02e000 comp rx=0 tx=0).stop 2024-04-25T01:22:55.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.707+0000 7f87be516640 1 -- 172.21.15.77:0/30859914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b80c8e30 msgr2=0x7f87b80cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.707+0000 7f87be516640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b80c8e30 0x7f87b80cc5d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f87a800e8a0 tx=0x7f87a800ed60 comp rx=0 tx=0).stop 2024-04-25T01:22:55.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.708+0000 7f87be516640 1 -- 172.21.15.77:0/30859914 shutdown_connections 2024-04-25T01:22:55.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.708+0000 7f87be516640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87a406be30 0x7f87a406e2f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.708+0000 7f87be516640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87b80ce090 0x7f87b80b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.708+0000 7f87be516640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87b80ca670 0x7f87b81f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.708+0000 7f87be516640 1 --2- 172.21.15.77:0/30859914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b80c8e30 0x7f87b80cc5d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.708+0000 7f87be516640 1 -- 172.21.15.77:0/30859914 >> 172.21.15.77:0/30859914 conn(0x7f87b80cfae0 msgr2=0x7f87b80cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:55.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.708+0000 7f87be516640 1 -- 172.21.15.77:0/30859914 shutdown_connections 2024-04-25T01:22:55.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.708+0000 7f87be516640 1 -- 172.21.15.77:0/30859914 wait complete. 2024-04-25T01:22:55.703 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T01:22:55.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.825+0000 7f3063872640 1 Processor -- start 2024-04-25T01:22:55.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.826+0000 7f3063872640 1 -- start start 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.826+0000 7f3063872640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305c0b2170 0x7f305c161f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.826+0000 7f3063872640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f305c0b2e80 0x7f305c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.826+0000 7f3063872640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f305c0cc260 con 0x7f305c0b2930 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.826+0000 7f3063872640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f305c162bc0 con 0x7f305c0b2e80 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.826+0000 7f3063872640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f305c162e10 con 0x7f305c0b2170 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.827+0000 7f3061e0f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f305c0b2e80 0x7f305c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.827+0000 7f3060e0d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f305c0b2930 0x7f305c1624b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36178/0 (socket says 172.21.15.77:36178) 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.827+0000 7f3060e0d640 1 -- 172.21.15.77:0/338662581 learned_addr learned my addr 172.21.15.77:0/338662581 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.827+0000 7f306160e640 1 --2- 172.21.15.77:0/338662581 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305c0b2170 0x7f305c161f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.827+0000 7f3061e0f640 1 -- 172.21.15.77:0/338662581 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305c0b2170 msgr2=0x7f305c161f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.827+0000 7f3061e0f640 1 --2- 172.21.15.77:0/338662581 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305c0b2170 0x7f305c161f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.827+0000 7f3061e0f640 1 -- 172.21.15.77:0/338662581 >> v1:172.21.15.77:6789/0 conn(0x7f305c0b2930 legacy=0x7f305c1624b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.827+0000 7f3061e0f640 1 -- 172.21.15.77:0/338662581 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f305c0d0b30 con 0x7f305c0b2e80 2024-04-25T01:22:55.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.827+0000 7f3053fff640 1 -- 172.21.15.77:0/338662581 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1456748561 0 0) 0x7f305c0cc260 con 0x7f305c0b2930 2024-04-25T01:22:55.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3061e0f640 1 --2- 172.21.15.77:0/338662581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f305c0b2e80 0x7f305c06c7e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3054009d90 tx=0x7f305402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c00626c8a7ab9b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:55.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3053fff640 1 -- 172.21.15.77:0/338662581 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3054007740 con 0x7f305c0b2e80 2024-04-25T01:22:55.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3053fff640 1 -- 172.21.15.77:0/338662581 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30540078e0 con 0x7f305c0b2e80 2024-04-25T01:22:55.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3053fff640 1 -- 172.21.15.77:0/338662581 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3054007cb0 con 0x7f305c0b2e80 2024-04-25T01:22:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3063872640 1 -- 172.21.15.77:0/338662581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f305c0b2e80 msgr2=0x7f305c06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3063872640 1 --2- 172.21.15.77:0/338662581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f305c0b2e80 0x7f305c06c7e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3054009d90 tx=0x7f305402a660 comp rx=0 tx=0).stop 2024-04-25T01:22:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3063872640 1 -- 172.21.15.77:0/338662581 shutdown_connections 2024-04-25T01:22:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3063872640 1 --2- 172.21.15.77:0/338662581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f305c0b2e80 0x7f305c06c7e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3063872640 1 --2- 172.21.15.77:0/338662581 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305c0b2170 0x7f305c161f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3063872640 1 -- 172.21.15.77:0/338662581 >> 172.21.15.77:0/338662581 conn(0x7f305c0d1c10 msgr2=0x7f305c0d2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3063872640 1 -- 172.21.15.77:0/338662581 shutdown_connections 2024-04-25T01:22:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.828+0000 7f3063872640 1 -- 172.21.15.77:0/338662581 wait complete. 2024-04-25T01:22:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.829+0000 7f3063872640 1 Processor -- start 2024-04-25T01:22:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f3063872640 1 -- start start 2024-04-25T01:22:55.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f3063872640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305c0b2170 0x7f305c1515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f3063872640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f305c0b2930 0x7f305c14bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f3063872640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f305c0b2e80 0x7f305c0701b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f3063872640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f305c0b2550 con 0x7f305c0b2930 2024-04-25T01:22:55.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f3063872640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f305c0d10f0 con 0x7f305c0b2e80 2024-04-25T01:22:55.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f3063872640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f305c0b26d0 con 0x7f305c0b2170 2024-04-25T01:22:55.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f3060e0d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f305c0b2930 0x7f305c14bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f3060e0d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f305c0b2930 0x7f305c14bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48730/0 (socket says 172.21.15.77:48730) 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f3060e0d640 1 -- 172.21.15.77:0/4029326185 learned_addr learned my addr 172.21.15.77:0/4029326185 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f306160e640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305c0b2170 0x7f305c1515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.830+0000 7f3061e0f640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f305c0b2e80 0x7f305c0701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f3060e0d640 1 -- 172.21.15.77:0/4029326185 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305c0b2170 msgr2=0x7f305c1515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f3060e0d640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305c0b2170 0x7f305c1515d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f3060e0d640 1 -- 172.21.15.77:0/4029326185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f305c0b2e80 msgr2=0x7f305c0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f3060e0d640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f305c0b2e80 0x7f305c0701b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f3060e0d640 1 -- 172.21.15.77:0/4029326185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f305c14c060 con 0x7f305c0b2930 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f306160e640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305c0b2170 0x7f305c1515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f3060e0d640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f305c0b2930 0x7f305c14bb20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f304c00c3f0 tx=0x7f304c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f3051ffb640 1 -- 172.21.15.77:0/4029326185 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304c0074e0 con 0x7f305c0b2930 2024-04-25T01:22:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f3051ffb640 1 -- 172.21.15.77:0/4029326185 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f304c0047e0 con 0x7f305c0b2930 2024-04-25T01:22:55.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f3063872640 1 -- 172.21.15.77:0/4029326185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f305c14ea00 con 0x7f305c0b2930 2024-04-25T01:22:55.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f3051ffb640 1 -- 172.21.15.77:0/4029326185 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304c004bd0 con 0x7f305c0b2930 2024-04-25T01:22:55.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.831+0000 7f3063872640 1 -- 172.21.15.77:0/4029326185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f305c0cabc0 con 0x7f305c0b2930 2024-04-25T01:22:55.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.832+0000 7f3063872640 1 -- 172.21.15.77:0/4029326185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f305c0ca710 con 0x7f305c0b2930 2024-04-25T01:22:55.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.832+0000 7f3051ffb640 1 -- 172.21.15.77:0/4029326185 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f304c011070 con 0x7f305c0b2930 2024-04-25T01:22:55.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.833+0000 7f3051ffb640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f304006be30 0x7f304006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.833+0000 7f306160e640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f304006be30 0x7f304006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.833+0000 7f3051ffb640 1 -- 172.21.15.77:0/4029326185 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5619+0+0 (secure 0 0 0) 0x7f304c08a950 con 0x7f305c0b2930 2024-04-25T01:22:55.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.833+0000 7f306160e640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f304006be30 0x7f304006e2f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3058006fd0 tx=0x7f3058008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:55.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:55.837+0000 7f3051ffb640 1 -- 172.21.15.77:0/4029326185 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f304c0593e0 con 0x7f305c0b2930 2024-04-25T01:22:55.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.010+0000 7f3063872640 1 -- 172.21.15.77:0/4029326185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f305c068fb0 con 0x7f305c0b2930 2024-04-25T01:22:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.713+0000 7f3051ffb640 1 -- 172.21.15.77:0/4029326185 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) ==== 170+0+0 (secure 0 0 0) 0x7f304c058ce0 con 0x7f305c0b2930 2024-04-25T01:22:56.695 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-04-25T01:22:56.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 -- 172.21.15.77:0/4029326185 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f304006be30 msgr2=0x7f304006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f304006be30 0x7f304006e2f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3058006fd0 tx=0x7f3058008040 comp rx=0 tx=0).stop 2024-04-25T01:22:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 -- 172.21.15.77:0/4029326185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f305c0b2930 msgr2=0x7f305c14bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f305c0b2930 0x7f305c14bb20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f304c00c3f0 tx=0x7f304c00c8b0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 -- 172.21.15.77:0/4029326185 shutdown_connections 2024-04-25T01:22:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f304006be30 0x7f304006e2f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f305c0b2930 0x7f305c14bb20 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f305c0b2e80 0x7f305c0701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 --2- 172.21.15.77:0/4029326185 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305c0b2170 0x7f305c1515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 -- 172.21.15.77:0/4029326185 >> 172.21.15.77:0/4029326185 conn(0x7f305c0d1c10 msgr2=0x7f305c0d1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 -- 172.21.15.77:0/4029326185 shutdown_connections 2024-04-25T01:22:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.716+0000 7f3063872640 1 -- 172.21.15.77:0/4029326185 wait complete. 2024-04-25T01:22:56.721 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T01:22:56.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.843+0000 7f0ac97a5640 1 Processor -- start 2024-04-25T01:22:56.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.843+0000 7f0ac97a5640 1 -- start start 2024-04-25T01:22:56.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.844+0000 7f0ac97a5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac4158a70 0x7f0ac40b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.844+0000 7f0ac97a5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0ac40b37c0 0x7f0ac40b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.844+0000 7f0ac97a5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ac415c210 con 0x7f0ac415b520 2024-04-25T01:22:56.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.844+0000 7f0ac97a5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0ac40b2ce0 con 0x7f0ac40b37c0 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.844+0000 7f0ac97a5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac40b2fc0 con 0x7f0ac4158a70 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.845+0000 7f0ac2ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac4158a70 0x7f0ac40b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.845+0000 7f0ac27fc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0ac40b37c0 0x7f0ac40b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.845+0000 7f0ac27fc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0ac40b37c0 0x7f0ac40b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:35172/0 (socket says 172.21.15.77:35172) 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.845+0000 7f0ac37fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0ac415b520 0x7f0ac415b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:59530/0 (socket says 172.21.15.77:59530) 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.845+0000 7f0ac2ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac4158a70 0x7f0ac40b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:53178/0 (socket says 172.21.15.77:53178) 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.845+0000 7f0ac27fc640 1 -- 172.21.15.77:0/971893899 learned_addr learned my addr 172.21.15.77:0/971893899 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.845+0000 7f0ac27fc640 1 -- 172.21.15.77:0/971893899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac4158a70 msgr2=0x7f0ac40b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.845+0000 7f0ac27fc640 1 --2- 172.21.15.77:0/971893899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac4158a70 0x7f0ac40b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.845+0000 7f0ac27fc640 1 -- 172.21.15.77:0/971893899 >> v1:172.21.15.121:6789/0 conn(0x7f0ac415b520 legacy=0x7f0ac415b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.845+0000 7f0ac27fc640 1 -- 172.21.15.77:0/971893899 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac40b32a0 con 0x7f0ac40b37c0 2024-04-25T01:22:56.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.845+0000 7f0ac2ffd640 1 --2- 172.21.15.77:0/971893899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac4158a70 0x7f0ac40b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:56.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.846+0000 7f0ac27fc640 1 --2- 172.21.15.77:0/971893899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac40b37c0 0x7f0ac40b4600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009870 tx=0x7f0ab802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7724804a9e34f77d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:56.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.846+0000 7f0ac1ffb640 1 -- 172.21.15.77:0/971893899 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8037070 con 0x7f0ac40b37c0 2024-04-25T01:22:56.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.846+0000 7f0ac1ffb640 1 -- 172.21.15.77:0/971893899 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ab8032440 con 0x7f0ac40b37c0 2024-04-25T01:22:56.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.846+0000 7f0ac1ffb640 1 -- 172.21.15.77:0/971893899 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8032810 con 0x7f0ac40b37c0 2024-04-25T01:22:56.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.846+0000 7f0ac97a5640 1 -- 172.21.15.77:0/971893899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac40b37c0 msgr2=0x7f0ac40b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.846+0000 7f0ac97a5640 1 --2- 172.21.15.77:0/971893899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac40b37c0 0x7f0ac40b4600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009870 tx=0x7f0ab802a270 comp rx=0 tx=0).stop 2024-04-25T01:22:56.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.847+0000 7f0ac97a5640 1 -- 172.21.15.77:0/971893899 shutdown_connections 2024-04-25T01:22:56.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.847+0000 7f0ac97a5640 1 --2- 172.21.15.77:0/971893899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac40b37c0 0x7f0ac40b4600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.847+0000 7f0ac97a5640 1 --2- 172.21.15.77:0/971893899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac4158a70 0x7f0ac40b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.847+0000 7f0ac97a5640 1 -- 172.21.15.77:0/971893899 >> 172.21.15.77:0/971893899 conn(0x7f0ac406b720 msgr2=0x7f0ac40b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:56.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.847+0000 7f0ac97a5640 1 -- 172.21.15.77:0/971893899 shutdown_connections 2024-04-25T01:22:56.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.847+0000 7f0ac97a5640 1 -- 172.21.15.77:0/971893899 wait complete. 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.848+0000 7f0ac97a5640 1 Processor -- start 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.848+0000 7f0ac97a5640 1 -- start start 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac97a5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac40b37c0 0x7f0ac40cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac97a5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac4158a70 0x7f0ac40a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac97a5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac415b520 0x7f0ac40aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac97a5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac406cb70 con 0x7f0ac4158a70 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac97a5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac406c9f0 con 0x7f0ac40b37c0 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac97a5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac406ccf0 con 0x7f0ac415b520 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac4158a70 0x7f0ac40a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac4158a70 0x7f0ac40a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44000/0 (socket says 172.21.15.77:44000) 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac27fc640 1 -- 172.21.15.77:0/4112979806 learned_addr learned my addr 172.21.15.77:0/4112979806 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:56.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac2ffd640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac40b37c0 0x7f0ac40cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac37fe640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac415b520 0x7f0ac40aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac27fc640 1 -- 172.21.15.77:0/4112979806 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac415b520 msgr2=0x7f0ac40aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac27fc640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac415b520 0x7f0ac40aeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac27fc640 1 -- 172.21.15.77:0/4112979806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac40b37c0 msgr2=0x7f0ac40cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac27fc640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac40b37c0 0x7f0ac40cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac27fc640 1 -- 172.21.15.77:0/4112979806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac40cda10 con 0x7f0ac4158a70 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac37fe640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac415b520 0x7f0ac40aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac27fc640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac4158a70 0x7f0ac40a9900 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8000c00 tx=0x7f0ab80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0aab7fe640 1 -- 172.21.15.77:0/4112979806 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8037040 con 0x7f0ac4158a70 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0aab7fe640 1 -- 172.21.15.77:0/4112979806 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ab8042070 con 0x7f0ac4158a70 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac97a5640 1 -- 172.21.15.77:0/4112979806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac40cdcf0 con 0x7f0ac4158a70 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0aab7fe640 1 -- 172.21.15.77:0/4112979806 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8030880 con 0x7f0ac4158a70 2024-04-25T01:22:56.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.849+0000 7f0ac97a5640 1 -- 172.21.15.77:0/4112979806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac40ce2b0 con 0x7f0ac4158a70 2024-04-25T01:22:56.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.850+0000 7f0ac97a5640 1 -- 172.21.15.77:0/4112979806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ac4068fb0 con 0x7f0ac4158a70 2024-04-25T01:22:56.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.851+0000 7f0aab7fe640 1 -- 172.21.15.77:0/4112979806 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0ab8030a20 con 0x7f0ac4158a70 2024-04-25T01:22:56.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.851+0000 7f0aab7fe640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ab006be60 0x7f0ab006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.852+0000 7f0ac2ffd640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ab006be60 0x7f0ab006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.852+0000 7f0aab7fe640 1 -- 172.21.15.77:0/4112979806 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6001+0+0 (secure 0 0 0) 0x7f0ab80ab720 con 0x7f0ac4158a70 2024-04-25T01:22:56.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.852+0000 7f0ac2ffd640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ab006be60 0x7f0ab006e320 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0abc009560 tx=0x7f0abc009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:56.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:56.856+0000 7f0aab7fe640 1 -- 172.21.15.77:0/4112979806 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0ab8079fa0 con 0x7f0ac4158a70 2024-04-25T01:22:57.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.064+0000 7f0ac97a5640 1 -- 172.21.15.77:0/4112979806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0ac40cd700 con 0x7f0ac4158a70 2024-04-25T01:22:57.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.730+0000 7f0aab7fe640 1 -- 172.21.15.77:0/4112979806 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) ==== 172+0+0 (secure 0 0 0) 0x7f0ab80798a0 con 0x7f0ac4158a70 2024-04-25T01:22:57.713 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 5 and data pool 6 2024-04-25T01:22:57.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.738+0000 7f0ac97a5640 1 -- 172.21.15.77:0/4112979806 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ab006be60 msgr2=0x7f0ab006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.738+0000 7f0ac97a5640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ab006be60 0x7f0ab006e320 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0abc009560 tx=0x7f0abc009190 comp rx=0 tx=0).stop 2024-04-25T01:22:57.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.738+0000 7f0ac97a5640 1 -- 172.21.15.77:0/4112979806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac4158a70 msgr2=0x7f0ac40a9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.738+0000 7f0ac97a5640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac4158a70 0x7f0ac40a9900 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8000c00 tx=0x7f0ab80046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.739+0000 7f0ac97a5640 1 -- 172.21.15.77:0/4112979806 shutdown_connections 2024-04-25T01:22:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.739+0000 7f0ac97a5640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0ab006be60 0x7f0ab006e320 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.739+0000 7f0ac97a5640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac4158a70 0x7f0ac40a9900 secure :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8000c00 tx=0x7f0ab80046c0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.739+0000 7f0ac97a5640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac415b520 0x7f0ac40aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.739+0000 7f0ac97a5640 1 --2- 172.21.15.77:0/4112979806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac40b37c0 0x7f0ac40cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.739+0000 7f0ac97a5640 1 -- 172.21.15.77:0/4112979806 >> 172.21.15.77:0/4112979806 conn(0x7f0ac406b720 msgr2=0x7f0ac41593e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.740+0000 7f0ac97a5640 1 -- 172.21.15.77:0/4112979806 shutdown_connections 2024-04-25T01:22:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.741+0000 7f0ac97a5640 1 -- 172.21.15.77:0/4112979806 wait complete. 2024-04-25T01:22:57.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:57.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.858+0000 7f8ff93c8640 1 Processor -- start 2024-04-25T01:22:57.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.858+0000 7f8ff93c8640 1 -- start start 2024-04-25T01:22:57.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff93c8640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8ff4143940 0x7f8ff406e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff93c8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8ff415f400 0x7f8ff406eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff93c8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ff4070e60 con 0x7f8ff4143d20 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff93c8640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f8ff4071140 con 0x7f8ff415f400 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff93c8640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8ff4071420 con 0x7f8ff4143940 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8ff415f400 0x7f8ff406eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8ff415f400 0x7f8ff406eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44002/0 (socket says 172.21.15.77:44002) 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff37fe640 1 -- 172.21.15.77:0/455944144 learned_addr learned my addr 172.21.15.77:0/455944144 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff2ffd640 1 --2- 172.21.15.77:0/455944144 >> v2:172.21.15.121:3300/0 conn(0x7f8ff4143940 0x7f8ff406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff37fe640 1 -- 172.21.15.77:0/455944144 >> v2:172.21.15.121:3300/0 conn(0x7f8ff4143940 msgr2=0x7f8ff406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff37fe640 1 --2- 172.21.15.77:0/455944144 >> v2:172.21.15.121:3300/0 conn(0x7f8ff4143940 0x7f8ff406e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff37fe640 1 -- 172.21.15.77:0/455944144 >> v1:172.21.15.121:6789/0 conn(0x7f8ff4143d20 legacy=0x7f8ff415ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.859+0000 7f8ff37fe640 1 -- 172.21.15.77:0/455944144 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff4071700 con 0x7f8ff415f400 2024-04-25T01:22:57.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.860+0000 7f8ff37fe640 1 --2- 172.21.15.77:0/455944144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff415f400 0x7f8ff406eb90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009870 tx=0x7f8fec02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8f4af888cb331b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:57.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.860+0000 7f8ff1ffb640 1 -- 172.21.15.77:0/455944144 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec037070 con 0x7f8ff415f400 2024-04-25T01:22:57.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.860+0000 7f8ff1ffb640 1 -- 172.21.15.77:0/455944144 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fec032440 con 0x7f8ff415f400 2024-04-25T01:22:57.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.860+0000 7f8ff1ffb640 1 -- 172.21.15.77:0/455944144 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec032810 con 0x7f8ff415f400 2024-04-25T01:22:57.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.861+0000 7f8ff93c8640 1 -- 172.21.15.77:0/455944144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff415f400 msgr2=0x7f8ff406eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.861+0000 7f8ff93c8640 1 --2- 172.21.15.77:0/455944144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff415f400 0x7f8ff406eb90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009870 tx=0x7f8fec02a2a0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.861+0000 7f8ff93c8640 1 -- 172.21.15.77:0/455944144 shutdown_connections 2024-04-25T01:22:57.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.861+0000 7f8ff93c8640 1 --2- 172.21.15.77:0/455944144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff415f400 0x7f8ff406eb90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.861+0000 7f8ff93c8640 1 --2- 172.21.15.77:0/455944144 >> v2:172.21.15.121:3300/0 conn(0x7f8ff4143940 0x7f8ff406e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.861+0000 7f8ff93c8640 1 -- 172.21.15.77:0/455944144 >> 172.21.15.77:0/455944144 conn(0x7f8ff40b1820 msgr2=0x7f8ff40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:57.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.861+0000 7f8ff93c8640 1 -- 172.21.15.77:0/455944144 shutdown_connections 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.861+0000 7f8ff93c8640 1 -- 172.21.15.77:0/455944144 wait complete. 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.862+0000 7f8ff93c8640 1 Processor -- start 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.862+0000 7f8ff93c8640 1 -- start start 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff93c8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ff4143940 0x7f8ff412f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff93c8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff4143d20 0x7f8ff4141c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff93c8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff415f400 0x7f8ff41421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff93c8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff406d910 con 0x7f8ff4143d20 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff93c8640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff406d790 con 0x7f8ff415f400 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff93c8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8ff406da90 con 0x7f8ff4143940 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff4143d20 0x7f8ff4141c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff4143d20 0x7f8ff4141c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44016/0 (socket says 172.21.15.77:44016) 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff27fc640 1 -- 172.21.15.77:0/2784055128 learned_addr learned my addr 172.21.15.77:0/2784055128 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff37fe640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff415f400 0x7f8ff41421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff2ffd640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ff4143940 0x7f8ff412f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff27fc640 1 -- 172.21.15.77:0/2784055128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ff4143940 msgr2=0x7f8ff412f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff27fc640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ff4143940 0x7f8ff412f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff27fc640 1 -- 172.21.15.77:0/2784055128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff415f400 msgr2=0x7f8ff41421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff27fc640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff415f400 0x7f8ff41421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff27fc640 1 -- 172.21.15.77:0/2784055128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff4155280 con 0x7f8ff4143d20 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.863+0000 7f8ff27fc640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff4143d20 0x7f8ff4141c90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800c430 tx=0x7f8fe800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.864+0000 7f8fdb7fe640 1 -- 172.21.15.77:0/2784055128 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe80052d0 con 0x7f8ff4143d20 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.864+0000 7f8fdb7fe640 1 -- 172.21.15.77:0/2784055128 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fe8005490 con 0x7f8ff4143d20 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.864+0000 7f8fdb7fe640 1 -- 172.21.15.77:0/2784055128 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8007260 con 0x7f8ff4143d20 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.864+0000 7f8ff93c8640 1 -- 172.21.15.77:0/2784055128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff41544c0 con 0x7f8ff4143d20 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.864+0000 7f8ff93c8640 1 -- 172.21.15.77:0/2784055128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff4154a80 con 0x7f8ff4143d20 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.865+0000 7f8ff93c8640 1 -- 172.21.15.77:0/2784055128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff4068fb0 con 0x7f8ff4143d20 2024-04-25T01:22:57.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.865+0000 7f8fdb7fe640 1 -- 172.21.15.77:0/2784055128 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8fe8007400 con 0x7f8ff4143d20 2024-04-25T01:22:57.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.866+0000 7f8fdb7fe640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8fe006be30 0x7f8fe006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.866+0000 7f8ff2ffd640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8fe006be30 0x7f8fe006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.866+0000 7f8fdb7fe640 1 -- 172.21.15.77:0/2784055128 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f8fe808a480 con 0x7f8ff4143d20 2024-04-25T01:22:57.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.866+0000 7f8ff2ffd640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8fe006be30 0x7f8fe006e2f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009fd0 tx=0x7f8fe400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:57.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:57.870+0000 7f8fdb7fe640 1 -- 172.21.15.77:0/2784055128 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8fe8058d30 con 0x7f8ff4143d20 2024-04-25T01:22:58.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.045+0000 7f8ff93c8640 1 -- 172.21.15.77:0/2784055128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ff415f1b0 con 0x7f8ff4143d20 2024-04-25T01:22:58.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.047+0000 7f8fdb7fe640 1 -- 172.21.15.77:0/2784055128 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16955 (secure 0 0 0) 0x7f8fe8058630 con 0x7f8ff4143d20 2024-04-25T01:22:58.029 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:58.029 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:57.066604+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.457887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.029883+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:58.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.048+0000 7f8ff93c8640 1 -- 172.21.15.77:0/2784055128 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8fe006be30 msgr2=0x7f8fe006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.048+0000 7f8ff93c8640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8fe006be30 0x7f8fe006e2f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009fd0 tx=0x7f8fe400b040 comp rx=0 tx=0).stop 2024-04-25T01:22:58.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.048+0000 7f8ff93c8640 1 -- 172.21.15.77:0/2784055128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff4143d20 msgr2=0x7f8ff4141c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.049+0000 7f8ff93c8640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff4143d20 0x7f8ff4141c90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800c430 tx=0x7f8fe800c8f0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.049+0000 7f8ff93c8640 1 -- 172.21.15.77:0/2784055128 shutdown_connections 2024-04-25T01:22:58.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.049+0000 7f8ff93c8640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8fe006be30 0x7f8fe006e2f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.049+0000 7f8ff93c8640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff4143d20 0x7f8ff4141c90 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.049+0000 7f8ff93c8640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff415f400 0x7f8ff41421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.049+0000 7f8ff93c8640 1 --2- 172.21.15.77:0/2784055128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ff4143940 0x7f8ff412f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.049+0000 7f8ff93c8640 1 -- 172.21.15.77:0/2784055128 >> 172.21.15.77:0/2784055128 conn(0x7f8ff40b1820 msgr2=0x7f8ff41593c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:58.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.049+0000 7f8ff93c8640 1 -- 172.21.15.77:0/2784055128 shutdown_connections 2024-04-25T01:22:58.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.049+0000 7f8ff93c8640 1 -- 172.21.15.77:0/2784055128 wait complete. 2024-04-25T01:22:58.043 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:58.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.158+0000 7f8b75893640 1 Processor -- start 2024-04-25T01:22:58.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.158+0000 7f8b75893640 1 -- start start 2024-04-25T01:22:58.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.159+0000 7f8b75893640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8b70143d20 0x7f8b7015ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.159+0000 7f8b75893640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b70070e60 con 0x7f8b70143940 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.159+0000 7f8b75893640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b70071140 con 0x7f8b7006ed60 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.159+0000 7f8b75893640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8b70071420 con 0x7f8b70143d20 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.159+0000 7f8b6effd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8b70143940 0x7f8b7006e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58458/0 (socket says 172.21.15.77:58458) 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.159+0000 7f8b6effd640 1 -- 172.21.15.77:0/1812661943 learned_addr learned my addr 172.21.15.77:0/1812661943 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.159+0000 7f8b6e7fc640 1 --2- 172.21.15.77:0/1812661943 >> v2:172.21.15.121:3300/0 conn(0x7f8b70143d20 0x7f8b7015ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.159+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806414351 0 0) 0x7f8b70070e60 con 0x7f8b70143940 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.159+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b58003520 con 0x7f8b70143940 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.159+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1264656277 0 0) 0x7f8b58003520 con 0x7f8b70143940 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 >> v2:172.21.15.121:3300/0 conn(0x7f8b70143d20 msgr2=0x7f8b7015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 --2- 172.21.15.77:0/1812661943 >> v2:172.21.15.121:3300/0 conn(0x7f8b70143d20 0x7f8b7015ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 >> v1:172.21.15.121:6789/0 conn(0x7f8b7006ed60 legacy=0x7f8b7015f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b70071700 con 0x7f8b70143940 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f8b600019f0 con 0x7f8b70143940 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 >> v1:172.21.15.77:6789/0 conn(0x7f8b70143940 legacy=0x7f8b7006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 --2- 172.21.15.77:0/1812661943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b580039e0 0x7f8b58003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 --2- 172.21.15.77:0/1812661943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 0x7f8b58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 --2- 172.21.15.77:0/1812661943 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b580051c0 0x7f8b5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8b70071420 con 0x7f8b58004800 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5800c090 con 0x7f8b580039e0 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8b5800c290 con 0x7f8b580051c0 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6e7fc640 1 --2- 172.21.15.77:0/1812661943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 0x7f8b58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6effd640 1 --2- 172.21.15.77:0/1812661943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b580039e0 0x7f8b58003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6f7fe640 1 --2- 172.21.15.77:0/1812661943 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b580051c0 0x7f8b5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6e7fc640 1 -- 172.21.15.77:0/1812661943 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b580051c0 msgr2=0x7f8b5800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:58.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6e7fc640 1 --2- 172.21.15.77:0/1812661943 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b580051c0 0x7f8b5800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6e7fc640 1 -- 172.21.15.77:0/1812661943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b580039e0 msgr2=0x7f8b58003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:58.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6e7fc640 1 --2- 172.21.15.77:0/1812661943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b580039e0 0x7f8b58003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6f7fe640 1 -- 172.21.15.77:0/1812661943 reap_dead start 2024-04-25T01:22:58.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.160+0000 7f8b6e7fc640 1 -- 172.21.15.77:0/1812661943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b58003520 con 0x7f8b58004800 2024-04-25T01:22:58.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.161+0000 7f8b6e7fc640 1 --2- 172.21.15.77:0/1812661943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 0x7f8b58004c80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009ac0 tx=0x7f8b58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda9f9916ddbc10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.161+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64036070 con 0x7f8b58004800 2024-04-25T01:22:58.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.161+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b64031400 con 0x7f8b58004800 2024-04-25T01:22:58.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.161+0000 7f8b6dffb640 1 -- 172.21.15.77:0/1812661943 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64009690 con 0x7f8b58004800 2024-04-25T01:22:58.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.161+0000 7f8b75893640 1 -- 172.21.15.77:0/1812661943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 msgr2=0x7f8b58004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.161+0000 7f8b75893640 1 --2- 172.21.15.77:0/1812661943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 0x7f8b58004c80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009ac0 tx=0x7f8b58005830 comp rx=0 tx=0).stop 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.161+0000 7f8b75893640 1 -- 172.21.15.77:0/1812661943 shutdown_connections 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.161+0000 7f8b75893640 1 --2- 172.21.15.77:0/1812661943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 0x7f8b58004c80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.161+0000 7f8b75893640 1 -- 172.21.15.77:0/1812661943 >> 172.21.15.77:0/1812661943 conn(0x7f8b700b1820 msgr2=0x7f8b700b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.161+0000 7f8b75893640 1 -- 172.21.15.77:0/1812661943 shutdown_connections 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.161+0000 7f8b75893640 1 -- 172.21.15.77:0/1812661943 wait complete. 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.162+0000 7f8b75893640 1 Processor -- start 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.163+0000 7f8b75893640 1 -- start start 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.163+0000 7f8b75893640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 0x7f8b7013e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.163+0000 7f8b75893640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b7013fd00 0x7f8b7013e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.163+0000 7f8b75893640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b70142570 0x7f8b701429f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.163+0000 7f8b75893640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7006e9e0 con 0x7f8b58004800 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.163+0000 7f8b75893640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7006e860 con 0x7f8b7013fd00 2024-04-25T01:22:58.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.163+0000 7f8b75893640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8b7006eb60 con 0x7f8b70142570 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.163+0000 7f8b6effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 0x7f8b7013e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b6effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 0x7f8b7013e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44036/0 (socket says 172.21.15.77:44036) 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b6effd640 1 -- 172.21.15.77:0/3020677514 learned_addr learned my addr 172.21.15.77:0/3020677514 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b6e7fc640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b7013fd00 0x7f8b7013e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b6f7fe640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b70142570 0x7f8b701429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b6effd640 1 -- 172.21.15.77:0/3020677514 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b70142570 msgr2=0x7f8b701429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b6effd640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b70142570 0x7f8b701429f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b6effd640 1 -- 172.21.15.77:0/3020677514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b7013fd00 msgr2=0x7f8b7013e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b6effd640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b7013fd00 0x7f8b7013e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b6effd640 1 -- 172.21.15.77:0/3020677514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7014d910 con 0x7f8b58004800 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b6e7fc640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b7013fd00 0x7f8b7013e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b6effd640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 0x7f8b7013e070 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8b60007970 tx=0x7f8b6000a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b57fff640 1 -- 172.21.15.77:0/3020677514 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60017020 con 0x7f8b58004800 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b57fff640 1 -- 172.21.15.77:0/3020677514 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b6000ada0 con 0x7f8b58004800 2024-04-25T01:22:58.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b75893640 1 -- 172.21.15.77:0/3020677514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b70140210 con 0x7f8b58004800 2024-04-25T01:22:58.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b57fff640 1 -- 172.21.15.77:0/3020677514 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60013610 con 0x7f8b58004800 2024-04-25T01:22:58.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.164+0000 7f8b75893640 1 -- 172.21.15.77:0/3020677514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b7014db80 con 0x7f8b58004800 2024-04-25T01:22:58.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.165+0000 7f8b75893640 1 -- 172.21.15.77:0/3020677514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b7014dd80 con 0x7f8b58004800 2024-04-25T01:22:58.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.165+0000 7f8b57fff640 1 -- 172.21.15.77:0/3020677514 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8b6001a020 con 0x7f8b58004800 2024-04-25T01:22:58.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.166+0000 7f8b57fff640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b5c06be30 0x7f8b5c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.166+0000 7f8b57fff640 1 -- 172.21.15.77:0/3020677514 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f8b6008b9b0 con 0x7f8b58004800 2024-04-25T01:22:58.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.166+0000 7f8b6e7fc640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b5c06be30 0x7f8b5c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.166+0000 7f8b6e7fc640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b5c06be30 0x7f8b5c06e2f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8b64033000 tx=0x7f8b64008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.170+0000 7f8b57fff640 1 -- 172.21.15.77:0/3020677514 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8b7014dd80 con 0x7f8b58004800 2024-04-25T01:22:58.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.346+0000 7f8b75893640 1 -- 172.21.15.77:0/3020677514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b70068fb0 con 0x7f8b58004800 2024-04-25T01:22:58.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.347+0000 7f8b57fff640 1 -- 172.21.15.77:0/3020677514 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16955 (secure 0 0 0) 0x7f8b60021020 con 0x7f8b58004800 2024-04-25T01:22:58.328 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:22:58.329 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:57.066604+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.457887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.029883+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 -- 172.21.15.77:0/3020677514 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b5c06be30 msgr2=0x7f8b5c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b5c06be30 0x7f8b5c06e2f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8b64033000 tx=0x7f8b64008030 comp rx=0 tx=0).stop 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 -- 172.21.15.77:0/3020677514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 msgr2=0x7f8b7013e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 0x7f8b7013e070 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8b60007970 tx=0x7f8b6000a6f0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 -- 172.21.15.77:0/3020677514 shutdown_connections 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8b5c06be30 0x7f8b5c06e2f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b70142570 0x7f8b701429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b7013fd00 0x7f8b7013e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 --2- 172.21.15.77:0/3020677514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b58004800 0x7f8b7013e070 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 -- 172.21.15.77:0/3020677514 >> 172.21.15.77:0/3020677514 conn(0x7f8b700b1820 msgr2=0x7f8b70072960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 -- 172.21.15.77:0/3020677514 shutdown_connections 2024-04-25T01:22:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.349+0000 7f8b75893640 1 -- 172.21.15.77:0/3020677514 wait complete. 2024-04-25T01:22:58.341 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T01:22:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.459+0000 7f0bab5e7640 1 Processor -- start 2024-04-25T01:22:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.460+0000 7f0bab5e7640 1 -- start start 2024-04-25T01:22:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.460+0000 7f0bab5e7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0ba40b9a90 0x7f0ba40d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.460+0000 7f0bab5e7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ba40cea40 con 0x7f0ba40d23d0 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.460+0000 7f0bab5e7640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ba40cece0 con 0x7f0ba40d27b0 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.460+0000 7f0bab5e7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0ba40cefc0 con 0x7f0ba40b9a90 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0ba9383640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0ba40d23d0 0x7f0ba40b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58464/0 (socket says 172.21.15.77:58464) 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0ba9383640 1 -- 172.21.15.77:0/2708107278 learned_addr learned my addr 172.21.15.77:0/2708107278 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0ba8b82640 1 --2- 172.21.15.77:0/2708107278 >> v2:172.21.15.121:3300/0 conn(0x7f0ba40b9a90 0x7f0ba40d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2729016593 0 0) 0x7f0ba40cea40 con 0x7f0ba40d23d0 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b90003520 con 0x7f0ba40d23d0 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 775289920 0 0) 0x7f0b90003520 con 0x7f0ba40d23d0 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 >> v2:172.21.15.121:3300/0 conn(0x7f0ba40b9a90 msgr2=0x7f0ba40d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 --2- 172.21.15.77:0/2708107278 >> v2:172.21.15.121:3300/0 conn(0x7f0ba40b9a90 0x7f0ba40d4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 >> v1:172.21.15.121:6789/0 conn(0x7f0ba40d27b0 legacy=0x7f0ba40b6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba40cf2a0 con 0x7f0ba40d23d0 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f0b94001690 con 0x7f0ba40d23d0 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 >> v1:172.21.15.77:6789/0 conn(0x7f0ba40d23d0 legacy=0x7f0ba40b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 --2- 172.21.15.77:0/2708107278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 0x7f0b90003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 --2- 172.21.15.77:0/2708107278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b900043b0 0x7f0b90004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 --2- 172.21.15.77:0/2708107278 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b90005080 0x7f0b9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba40cefc0 con 0x7f0b900039b0 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9000bf30 con 0x7f0b900043b0 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0b9000c130 con 0x7f0b90005080 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0ba9383640 1 --2- 172.21.15.77:0/2708107278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 0x7f0b90003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0ba8b82640 1 --2- 172.21.15.77:0/2708107278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b900043b0 0x7f0b90004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0ba9b84640 1 --2- 172.21.15.77:0/2708107278 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b90005080 0x7f0b9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0ba9383640 1 -- 172.21.15.77:0/2708107278 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b90005080 msgr2=0x7f0b9000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.461+0000 7f0ba9383640 1 --2- 172.21.15.77:0/2708107278 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b90005080 0x7f0b9000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.462+0000 7f0ba9383640 1 -- 172.21.15.77:0/2708107278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b900043b0 msgr2=0x7f0b90004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.462+0000 7f0ba9383640 1 --2- 172.21.15.77:0/2708107278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b900043b0 0x7f0b90004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.462+0000 7f0ba9383640 1 -- 172.21.15.77:0/2708107278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b90003520 con 0x7f0b900039b0 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.462+0000 7f0ba9b84640 1 -- 172.21.15.77:0/2708107278 reap_dead start 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.462+0000 7f0ba9383640 1 --2- 172.21.15.77:0/2708107278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 0x7f0b90003d90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40cec10 tx=0x7f0b90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65700d4829315f1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.462+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b94007ce0 con 0x7f0b900039b0 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.462+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b94005ad0 con 0x7f0b900039b0 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.462+0000 7f0b9bfff640 1 -- 172.21.15.77:0/2708107278 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b94007070 con 0x7f0b900039b0 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.463+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2708107278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 msgr2=0x7f0b90003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.463+0000 7f0bab5e7640 1 --2- 172.21.15.77:0/2708107278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 0x7f0b90003d90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40cec10 tx=0x7f0b90005830 comp rx=0 tx=0).stop 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.463+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2708107278 shutdown_connections 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.463+0000 7f0bab5e7640 1 --2- 172.21.15.77:0/2708107278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 0x7f0b90003d90 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.463+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2708107278 >> 172.21.15.77:0/2708107278 conn(0x7f0ba40d3e30 msgr2=0x7f0ba40d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:58.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.463+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2708107278 shutdown_connections 2024-04-25T01:22:58.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.463+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2708107278 wait complete. 2024-04-25T01:22:58.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.464+0000 7f0bab5e7640 1 Processor -- start 2024-04-25T01:22:58.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.464+0000 7f0bab5e7640 1 -- start start 2024-04-25T01:22:58.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0bab5e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 0x7f0ba415aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0bab5e7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ba41f7690 0x7f0ba41f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0bab5e7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ba4158750 0x7f0ba4158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0bab5e7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba40cfd80 con 0x7f0b900039b0 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0bab5e7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba40cf860 con 0x7f0ba4158750 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0bab5e7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba40d4920 con 0x7f0ba41f7690 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0ba9383640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 0x7f0ba415aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0ba9383640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 0x7f0ba415aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44046/0 (socket says 172.21.15.77:44046) 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0ba9383640 1 -- 172.21.15.77:0/2241276948 learned_addr learned my addr 172.21.15.77:0/2241276948 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0ba9383640 1 -- 172.21.15.77:0/2241276948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ba41f7690 msgr2=0x7f0ba41f7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0ba9383640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ba41f7690 0x7f0ba41f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0ba9383640 1 -- 172.21.15.77:0/2241276948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ba4158750 msgr2=0x7f0ba4158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0ba9383640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ba4158750 0x7f0ba4158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0ba9383640 1 -- 172.21.15.77:0/2241276948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba41599f0 con 0x7f0b900039b0 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0ba9383640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 0x7f0ba415aa50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400c300 tx=0x7f0ba40ceeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0b99ffb640 1 -- 172.21.15.77:0/2241276948 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b94040030 con 0x7f0b900039b0 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0b99ffb640 1 -- 172.21.15.77:0/2241276948 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b94000ed0 con 0x7f0b900039b0 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2241276948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba4159c90 con 0x7f0b900039b0 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.465+0000 7f0b99ffb640 1 -- 172.21.15.77:0/2241276948 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b94037040 con 0x7f0b900039b0 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.466+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2241276948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba415a250 con 0x7f0b900039b0 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.467+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2241276948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ba40ce620 con 0x7f0b900039b0 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.467+0000 7f0b99ffb640 1 -- 172.21.15.77:0/2241276948 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0b94043070 con 0x7f0b900039b0 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.467+0000 7f0b99ffb640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0b8806be60 0x7f0b8806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.468+0000 7f0b99ffb640 1 -- 172.21.15.77:0/2241276948 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f0b940aa310 con 0x7f0b900039b0 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.468+0000 7f0ba8b82640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0b8806be60 0x7f0b8806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.468+0000 7f0ba8b82640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0b8806be60 0x7f0b8806e320 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0006e80 tx=0x7f0ba0006df0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.472+0000 7f0b99ffb640 1 -- 172.21.15.77:0/2241276948 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0ba40ce620 con 0x7f0b900039b0 2024-04-25T01:22:58.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.682+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2241276948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0ba4068fb0 con 0x7f0b900039b0 2024-04-25T01:22:58.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.724+0000 7f0b99ffb640 1 -- 172.21.15.77:0/2241276948 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f0b9404a020 con 0x7f0b900039b0 2024-04-25T01:22:58.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2241276948 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0b8806be60 msgr2=0x7f0b8806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0b8806be60 0x7f0b8806e320 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0006e80 tx=0x7f0ba0006df0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2241276948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 msgr2=0x7f0ba415aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 0x7f0ba415aa50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400c300 tx=0x7f0ba40ceeb0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2241276948 shutdown_connections 2024-04-25T01:22:58.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f0b8806be60 0x7f0b8806e320 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ba4158750 0x7f0ba4158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ba41f7690 0x7f0ba41f7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 --2- 172.21.15.77:0/2241276948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b900039b0 0x7f0ba415aa50 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2241276948 >> 172.21.15.77:0/2241276948 conn(0x7f0ba40d3e30 msgr2=0x7f0ba40b58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:58.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2241276948 shutdown_connections 2024-04-25T01:22:58.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.727+0000 7f0bab5e7640 1 -- 172.21.15.77:0/2241276948 wait complete. 2024-04-25T01:22:58.720 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T01:22:58.721 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T01:22:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.839+0000 7f128c8f5640 1 Processor -- start 2024-04-25T01:22:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.839+0000 7f128c8f5640 1 -- start start 2024-04-25T01:22:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.840+0000 7f128c8f5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12880b9a90 0x7f12880d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.840+0000 7f128c8f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 0x7f12880d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.840+0000 7f128c8f5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12880cea40 con 0x7f12880d23d0 2024-04-25T01:22:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.840+0000 7f128c8f5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f12880cece0 con 0x7f12880b9a90 2024-04-25T01:22:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.840+0000 7f128c8f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12880cefc0 con 0x7f12880d27b0 2024-04-25T01:22:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.841+0000 7f1285d9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12880b9a90 0x7f12880d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.841+0000 7f1286d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 0x7f12880d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.841+0000 7f1285d9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12880b9a90 0x7f12880d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:35238/0 (socket says 172.21.15.77:35238) 2024-04-25T01:22:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.841+0000 7f1285d9b640 1 -- 172.21.15.77:0/628512643 learned_addr learned my addr 172.21.15.77:0/628512643 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.841+0000 7f1286d9d640 1 --2- 172.21.15.77:0/628512643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 0x7f12880d0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:53276/0 (socket says 172.21.15.77:53276) 2024-04-25T01:22:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.841+0000 7f128659c640 1 --1- 172.21.15.77:0/628512643 >> v1:172.21.15.121:6789/0 conn(0x7f12880d23d0 0x7f12880b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:59554/0 (socket says 172.21.15.77:59554) 2024-04-25T01:22:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.841+0000 7f1285d9b640 1 -- 172.21.15.77:0/628512643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 msgr2=0x7f12880d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.841+0000 7f1285d9b640 1 --2- 172.21.15.77:0/628512643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 0x7f12880d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.841+0000 7f1285d9b640 1 -- 172.21.15.77:0/628512643 >> v1:172.21.15.121:6789/0 conn(0x7f12880d23d0 legacy=0x7f12880b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.841+0000 7f1285d9b640 1 -- 172.21.15.77:0/628512643 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12880cf2a0 con 0x7f12880b9a90 2024-04-25T01:22:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.841+0000 7f1286d9d640 1 --2- 172.21.15.77:0/628512643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 0x7f12880d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.842+0000 7f1285d9b640 1 --2- 172.21.15.77:0/628512643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12880b9a90 0x7f12880d4920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f127c009870 tx=0x7f127c02a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86b1895148987a1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.842+0000 7f128559a640 1 -- 172.21.15.77:0/628512643 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127c037070 con 0x7f12880b9a90 2024-04-25T01:22:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.842+0000 7f128559a640 1 -- 172.21.15.77:0/628512643 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f127c02ae50 con 0x7f12880b9a90 2024-04-25T01:22:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.842+0000 7f128559a640 1 -- 172.21.15.77:0/628512643 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127c032860 con 0x7f12880b9a90 2024-04-25T01:22:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.842+0000 7f128c8f5640 1 -- 172.21.15.77:0/628512643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12880b9a90 msgr2=0x7f12880d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.842+0000 7f128c8f5640 1 --2- 172.21.15.77:0/628512643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12880b9a90 0x7f12880d4920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f127c009870 tx=0x7f127c02a300 comp rx=0 tx=0).stop 2024-04-25T01:22:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.843+0000 7f128c8f5640 1 -- 172.21.15.77:0/628512643 shutdown_connections 2024-04-25T01:22:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.843+0000 7f128c8f5640 1 --2- 172.21.15.77:0/628512643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 0x7f12880d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.843+0000 7f128c8f5640 1 --2- 172.21.15.77:0/628512643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12880b9a90 0x7f12880d4920 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.843+0000 7f128c8f5640 1 -- 172.21.15.77:0/628512643 >> 172.21.15.77:0/628512643 conn(0x7f12880d3e30 msgr2=0x7f12880d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.843+0000 7f128c8f5640 1 -- 172.21.15.77:0/628512643 shutdown_connections 2024-04-25T01:22:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.843+0000 7f128c8f5640 1 -- 172.21.15.77:0/628512643 wait complete. 2024-04-25T01:22:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.844+0000 7f128c8f5640 1 Processor -- start 2024-04-25T01:22:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.844+0000 7f128c8f5640 1 -- start start 2024-04-25T01:22:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128c8f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12880b9a90 0x7f128815e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128c8f5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12880d23d0 0x7f12880c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128c8f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 0x7f1288157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128c8f5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f12880cfd80 con 0x7f12880b9a90 2024-04-25T01:22:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128c8f5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f12880cf860 con 0x7f12880d23d0 2024-04-25T01:22:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128c8f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12880b6c80 con 0x7f12880d27b0 2024-04-25T01:22:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12880b9a90 0x7f128815e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12880b9a90 0x7f128815e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44054/0 (socket says 172.21.15.77:44054) 2024-04-25T01:22:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128659c640 1 -- 172.21.15.77:0/1302354884 learned_addr learned my addr 172.21.15.77:0/1302354884 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f1285d9b640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12880d23d0 0x7f12880c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f1286d9d640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 0x7f1288157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128659c640 1 -- 172.21.15.77:0/1302354884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 msgr2=0x7f1288157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128659c640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 0x7f1288157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128659c640 1 -- 172.21.15.77:0/1302354884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12880d23d0 msgr2=0x7f12880c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128659c640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12880d23d0 0x7f12880c9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128659c640 1 -- 172.21.15.77:0/1302354884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12881f7610 con 0x7f12880b9a90 2024-04-25T01:22:58.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f1285d9b640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12880d23d0 0x7f12880c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:58.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128659c640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12880b9a90 0x7f128815e770 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f127800c790 tx=0x7f127800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f1276ffd640 1 -- 172.21.15.77:0/1302354884 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1278017070 con 0x7f12880b9a90 2024-04-25T01:22:58.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f1276ffd640 1 -- 172.21.15.77:0/1302354884 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f127800a640 con 0x7f12880b9a90 2024-04-25T01:22:58.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128c8f5640 1 -- 172.21.15.77:0/1302354884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12881578d0 con 0x7f12880b9a90 2024-04-25T01:22:58.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f1276ffd640 1 -- 172.21.15.77:0/1302354884 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127800a9f0 con 0x7f12880b9a90 2024-04-25T01:22:58.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.845+0000 7f128c8f5640 1 -- 172.21.15.77:0/1302354884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12880ca4b0 con 0x7f12880b9a90 2024-04-25T01:22:58.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.846+0000 7f128c8f5640 1 -- 172.21.15.77:0/1302354884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1288068fb0 con 0x7f12880b9a90 2024-04-25T01:22:58.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.847+0000 7f1276ffd640 1 -- 172.21.15.77:0/1302354884 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f127801a070 con 0x7f12880b9a90 2024-04-25T01:22:58.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.847+0000 7f1276ffd640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f126806be30 0x7f126806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.847+0000 7f1285d9b640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f126806be30 0x7f126806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.847+0000 7f1276ffd640 1 -- 172.21.15.77:0/1302354884 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f127808a590 con 0x7f12880b9a90 2024-04-25T01:22:58.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.848+0000 7f1285d9b640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f126806be30 0x7f126806e2f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f127c034000 tx=0x7f127c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:58.852+0000 7f1276ffd640 1 -- 172.21.15.77:0/1302354884 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1278058770 con 0x7f12880b9a90 2024-04-25T01:22:59.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.061+0000 7f128c8f5640 1 -- 172.21.15.77:0/1302354884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f128808df50 con 0x7f12880b9a90 2024-04-25T01:22:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.735+0000 7f1276ffd640 1 -- 172.21.15.77:0/1302354884 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f127805ca70 con 0x7f12880b9a90 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 -- 172.21.15.77:0/1302354884 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f126806be30 msgr2=0x7f126806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f126806be30 0x7f126806e2f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f127c034000 tx=0x7f127c02e000 comp rx=0 tx=0).stop 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 -- 172.21.15.77:0/1302354884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12880b9a90 msgr2=0x7f128815e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12880b9a90 0x7f128815e770 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f127800c790 tx=0x7f127800cc50 comp rx=0 tx=0).stop 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 -- 172.21.15.77:0/1302354884 shutdown_connections 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f126806be30 0x7f126806e2f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12880d27b0 0x7f1288157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12880d23d0 0x7f12880c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 --2- 172.21.15.77:0/1302354884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12880b9a90 0x7f128815e770 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 -- 172.21.15.77:0/1302354884 >> 172.21.15.77:0/1302354884 conn(0x7f12880d3e30 msgr2=0x7f12880b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 -- 172.21.15.77:0/1302354884 shutdown_connections 2024-04-25T01:22:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.737+0000 7f128c8f5640 1 -- 172.21.15.77:0/1302354884 wait complete. 2024-04-25T01:22:59.731 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T01:22:59.732 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:59.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.852+0000 7fdee4a58640 1 Processor -- start 2024-04-25T01:22:59.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.852+0000 7fdee4a58640 1 -- start start 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdee4a58640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdee0145590 0x7fdee00b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdee4a58640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdee006e280 0x7fdee015fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdee4a58640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee00b4240 con 0x7fdee015f9b0 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdee4a58640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fdee0160830 con 0x7fdee006e280 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdee4a58640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fdee0160350 con 0x7fdee0145590 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdeded9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdee006e280 0x7fdee015fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdeded9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdee006e280 0x7fdee015fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44058/0 (socket says 172.21.15.77:44058) 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdeded9d640 1 -- 172.21.15.77:0/4033836646 learned_addr learned my addr 172.21.15.77:0/4033836646 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdede59c640 1 --2- 172.21.15.77:0/4033836646 >> v2:172.21.15.121:3300/0 conn(0x7fdee0145590 0x7fdee00b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdeded9d640 1 -- 172.21.15.77:0/4033836646 >> v2:172.21.15.121:3300/0 conn(0x7fdee0145590 msgr2=0x7fdee00b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdeded9d640 1 --2- 172.21.15.77:0/4033836646 >> v2:172.21.15.121:3300/0 conn(0x7fdee0145590 0x7fdee00b4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.853+0000 7fdeded9d640 1 -- 172.21.15.77:0/4033836646 >> v1:172.21.15.121:6789/0 conn(0x7fdee015f9b0 legacy=0x7fdee00ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:22:59.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.854+0000 7fdeded9d640 1 -- 172.21.15.77:0/4033836646 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee00b4f60 con 0x7fdee006e280 2024-04-25T01:22:59.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.854+0000 7fdeded9d640 1 --2- 172.21.15.77:0/4033836646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee006e280 0x7fdee015fe10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdee006c150 tx=0x7fded802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe75895fb34af53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:59.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.854+0000 7fdeddd9b640 1 -- 172.21.15.77:0/4033836646 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded8036070 con 0x7fdee006e280 2024-04-25T01:22:59.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.854+0000 7fdeddd9b640 1 -- 172.21.15.77:0/4033836646 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded8031600 con 0x7fdee006e280 2024-04-25T01:22:59.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.854+0000 7fdeddd9b640 1 -- 172.21.15.77:0/4033836646 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded80319d0 con 0x7fdee006e280 2024-04-25T01:22:59.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.855+0000 7fdee4a58640 1 -- 172.21.15.77:0/4033836646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee006e280 msgr2=0x7fdee015fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:59.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.855+0000 7fdee4a58640 1 --2- 172.21.15.77:0/4033836646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee006e280 0x7fdee015fe10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdee006c150 tx=0x7fded802a730 comp rx=0 tx=0).stop 2024-04-25T01:22:59.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.855+0000 7fdee4a58640 1 -- 172.21.15.77:0/4033836646 shutdown_connections 2024-04-25T01:22:59.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.855+0000 7fdee4a58640 1 --2- 172.21.15.77:0/4033836646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee006e280 0x7fdee015fe10 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.855+0000 7fdee4a58640 1 --2- 172.21.15.77:0/4033836646 >> v2:172.21.15.121:3300/0 conn(0x7fdee0145590 0x7fdee00b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.855+0000 7fdee4a58640 1 -- 172.21.15.77:0/4033836646 >> 172.21.15.77:0/4033836646 conn(0x7fdee00b1820 msgr2=0x7fdee00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:59.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.855+0000 7fdee4a58640 1 -- 172.21.15.77:0/4033836646 shutdown_connections 2024-04-25T01:22:59.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.855+0000 7fdee4a58640 1 -- 172.21.15.77:0/4033836646 wait complete. 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.856+0000 7fdee4a58640 1 Processor -- start 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.856+0000 7fdee4a58640 1 -- start start 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fdee4a58640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee006e280 0x7fdee015e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fdee4a58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee0145590 0x7fdee013d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fdee4a58640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee015f9b0 0x7fdee013d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fdee4a58640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdee00b3c80 con 0x7fdee0145590 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fdee4a58640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdee006cde0 con 0x7fdee006e280 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fdee4a58640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdee00b3f60 con 0x7fdee015f9b0 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fded7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee0145590 0x7fdee013d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fded7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee0145590 0x7fdee013d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44074/0 (socket says 172.21.15.77:44074) 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fded7fff640 1 -- 172.21.15.77:0/2730103798 learned_addr learned my addr 172.21.15.77:0/2730103798 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fdede59c640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee006e280 0x7fdee015e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fdeded9d640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee015f9b0 0x7fdee013d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fded7fff640 1 -- 172.21.15.77:0/2730103798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee015f9b0 msgr2=0x7fdee013d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fded7fff640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee015f9b0 0x7fdee013d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fded7fff640 1 -- 172.21.15.77:0/2730103798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee006e280 msgr2=0x7fdee015e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fded7fff640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee006e280 0x7fdee015e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fded7fff640 1 -- 172.21.15.77:0/2730103798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee01e6ec0 con 0x7fdee0145590 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fdede59c640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee006e280 0x7fdee015e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fdeded9d640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee015f9b0 0x7fdee013d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.857+0000 7fded7fff640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee0145590 0x7fdee013d250 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdec800c3c0 tx=0x7fdec800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.858+0000 7fded6ffd640 1 -- 172.21.15.77:0/2730103798 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec80052d0 con 0x7fdee0145590 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.858+0000 7fded6ffd640 1 -- 172.21.15.77:0/2730103798 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec8005490 con 0x7fdee0145590 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.858+0000 7fdee4a58640 1 -- 172.21.15.77:0/2730103798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee014c220 con 0x7fdee0145590 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.858+0000 7fded6ffd640 1 -- 172.21.15.77:0/2730103798 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8007260 con 0x7fdee0145590 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.858+0000 7fdee4a58640 1 -- 172.21.15.77:0/2730103798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee015e6b0 con 0x7fdee0145590 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.859+0000 7fdee4a58640 1 -- 172.21.15.77:0/2730103798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdee0068fb0 con 0x7fdee0145590 2024-04-25T01:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.859+0000 7fded6ffd640 1 -- 172.21.15.77:0/2730103798 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdec8007400 con 0x7fdee0145590 2024-04-25T01:22:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.859+0000 7fded6ffd640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdec006be60 0x7fdec006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.859+0000 7fded6ffd640 1 -- 172.21.15.77:0/2730103798 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fdec808a490 con 0x7fdee0145590 2024-04-25T01:22:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.859+0000 7fdede59c640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdec006be60 0x7fdec006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.860+0000 7fdede59c640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdec006be60 0x7fdec006e320 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fded0006fd0 tx=0x7fded0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:59.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:22:59.864+0000 7fded6ffd640 1 -- 172.21.15.77:0/2730103798 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdec8058c90 con 0x7fdee0145590 2024-04-25T01:23:00.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.073+0000 7fdee4a58640 1 -- 172.21.15.77:0/2730103798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdee006ea50 con 0x7fdee0145590 2024-04-25T01:23:00.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.074+0000 7fded6ffd640 1 -- 172.21.15.77:0/2730103798 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5187 (secure 0 0 0) 0x7fdec8058590 con 0x7fdee0145590 2024-04-25T01:23:00.055 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:00.055 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3139470828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3139470828},{"type":"v1","addr":"172.21.15.77:6835","nonce":3139470828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.066906+0000","modified":"2024-04-25T01:22:59.063205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5358":{"gid":5358,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2944320661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2944320661},{"type":"v1","addr":"172.21.15.77:6837","nonce":2944320661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5358,"qdb_cluster":[5358]},"id":2}]} 2024-04-25T01:23:00.056 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 16 2024-04-25T01:23:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 -- 172.21.15.77:0/2730103798 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdec006be60 msgr2=0x7fdec006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdec006be60 0x7fdec006e320 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fded0006fd0 tx=0x7fded0008040 comp rx=0 tx=0).stop 2024-04-25T01:23:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 -- 172.21.15.77:0/2730103798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee0145590 msgr2=0x7fdee013d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee0145590 0x7fdee013d250 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdec800c3c0 tx=0x7fdec800c880 comp rx=0 tx=0).stop 2024-04-25T01:23:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 -- 172.21.15.77:0/2730103798 shutdown_connections 2024-04-25T01:23:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fdec006be60 0x7fdec006e320 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee0145590 0x7fdee013d250 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee015f9b0 0x7fdee013d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 --2- 172.21.15.77:0/2730103798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee006e280 0x7fdee015e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 -- 172.21.15.77:0/2730103798 >> 172.21.15.77:0/2730103798 conn(0x7fdee00b1820 msgr2=0x7fdee00af770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 -- 172.21.15.77:0/2730103798 shutdown_connections 2024-04-25T01:23:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.076+0000 7fdee4a58640 1 -- 172.21.15.77:0/2730103798 wait complete. 2024-04-25T01:23:00.068 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:00.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.183+0000 7f41b68cd640 1 Processor -- start 2024-04-25T01:23:00.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.183+0000 7f41b68cd640 1 -- start start 2024-04-25T01:23:00.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b68cd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b0143d20 0x7f41b015ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b68cd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f41b006ed60 0x7f41b015f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b68cd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b015f770 con 0x7f41b0143940 2024-04-25T01:23:00.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b68cd640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f41b0070e60 con 0x7f41b006ed60 2024-04-25T01:23:00.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b68cd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f41b0071140 con 0x7f41b0143d20 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b4e6a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f41b006ed60 0x7f41b015f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b4e6a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f41b006ed60 0x7f41b015f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44088/0 (socket says 172.21.15.77:44088) 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b4e6a640 1 -- 172.21.15.77:0/1097348841 learned_addr learned my addr 172.21.15.77:0/1097348841 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41af7fe640 1 --2- 172.21.15.77:0/1097348841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b0143d20 0x7f41b015ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b4e6a640 1 -- 172.21.15.77:0/1097348841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b0143d20 msgr2=0x7f41b015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b4e6a640 1 --2- 172.21.15.77:0/1097348841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b0143d20 0x7f41b015ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b4e6a640 1 -- 172.21.15.77:0/1097348841 >> v1:172.21.15.121:6789/0 conn(0x7f41b0143940 legacy=0x7f41b006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.184+0000 7f41b4e6a640 1 -- 172.21.15.77:0/1097348841 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b0071420 con 0x7f41b006ed60 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.185+0000 7f41b4e6a640 1 --2- 172.21.15.77:0/1097348841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41b006ed60 0x7f41b015f230 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f41a800d180 tx=0x7f41a802c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a63781a873a3b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.185+0000 7f41aeffd640 1 -- 172.21.15.77:0/1097348841 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8009600 con 0x7f41b006ed60 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.185+0000 7f41aeffd640 1 -- 172.21.15.77:0/1097348841 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a8037070 con 0x7f41b006ed60 2024-04-25T01:23:00.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.185+0000 7f41aeffd640 1 -- 172.21.15.77:0/1097348841 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a80099c0 con 0x7f41b006ed60 2024-04-25T01:23:00.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.185+0000 7f41b68cd640 1 -- 172.21.15.77:0/1097348841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41b006ed60 msgr2=0x7f41b015f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.185+0000 7f41b68cd640 1 --2- 172.21.15.77:0/1097348841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41b006ed60 0x7f41b015f230 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f41a800d180 tx=0x7f41a802c960 comp rx=0 tx=0).stop 2024-04-25T01:23:00.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.186+0000 7f41b68cd640 1 -- 172.21.15.77:0/1097348841 shutdown_connections 2024-04-25T01:23:00.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.186+0000 7f41b68cd640 1 --2- 172.21.15.77:0/1097348841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41b006ed60 0x7f41b015f230 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.186+0000 7f41b68cd640 1 --2- 172.21.15.77:0/1097348841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b0143d20 0x7f41b015ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.186+0000 7f41b68cd640 1 -- 172.21.15.77:0/1097348841 >> 172.21.15.77:0/1097348841 conn(0x7f41b00b1820 msgr2=0x7f41b00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:00.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.186+0000 7f41b68cd640 1 -- 172.21.15.77:0/1097348841 shutdown_connections 2024-04-25T01:23:00.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.186+0000 7f41b68cd640 1 -- 172.21.15.77:0/1097348841 wait complete. 2024-04-25T01:23:00.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.187+0000 7f41b68cd640 1 Processor -- start 2024-04-25T01:23:00.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.187+0000 7f41b68cd640 1 -- start start 2024-04-25T01:23:00.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41b68cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41b006ed60 0x7f41b00acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41b68cd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41b0143940 0x7f41b015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41b68cd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b0143d20 0x7f41b00c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41b68cd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41b006d6c0 con 0x7f41b006ed60 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41b68cd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f41b006d540 con 0x7f41b0143940 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41b68cd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f41b006d840 con 0x7f41b0143d20 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41b006ed60 0x7f41b00acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41b006ed60 0x7f41b00acc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44096/0 (socket says 172.21.15.77:44096) 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41affff640 1 -- 172.21.15.77:0/542224044 learned_addr learned my addr 172.21.15.77:0/542224044 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41affff640 1 -- 172.21.15.77:0/542224044 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b0143d20 msgr2=0x7f41b00c5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41af7fe640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41b0143940 0x7f41b015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41b4e6a640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b0143d20 0x7f41b00c5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41affff640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b0143d20 0x7f41b00c5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41affff640 1 -- 172.21.15.77:0/542224044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41b0143940 msgr2=0x7f41b015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41affff640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41b0143940 0x7f41b015d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41affff640 1 -- 172.21.15.77:0/542224044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b00abb80 con 0x7f41b006ed60 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41af7fe640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41b0143940 0x7f41b015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:00.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41affff640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41b006ed60 0x7f41b00acc20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f419800c400 tx=0x7f419800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41acff9640 1 -- 172.21.15.77:0/542224044 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41980052d0 con 0x7f41b006ed60 2024-04-25T01:23:00.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41acff9640 1 -- 172.21.15.77:0/542224044 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4198005490 con 0x7f41b006ed60 2024-04-25T01:23:00.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41b68cd640 1 -- 172.21.15.77:0/542224044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b00abe60 con 0x7f41b006ed60 2024-04-25T01:23:00.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41acff9640 1 -- 172.21.15.77:0/542224044 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4198007260 con 0x7f41b006ed60 2024-04-25T01:23:00.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.188+0000 7f41b68cd640 1 -- 172.21.15.77:0/542224044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b00ac420 con 0x7f41b006ed60 2024-04-25T01:23:00.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.189+0000 7f41b68cd640 1 -- 172.21.15.77:0/542224044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b0068fb0 con 0x7f41b006ed60 2024-04-25T01:23:00.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.190+0000 7f41acff9640 1 -- 172.21.15.77:0/542224044 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4198007400 con 0x7f41b006ed60 2024-04-25T01:23:00.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.190+0000 7f41acff9640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f419006be30 0x7f419006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.190+0000 7f41af7fe640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f419006be30 0x7f419006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.190+0000 7f41acff9640 1 -- 172.21.15.77:0/542224044 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f419808a5c0 con 0x7f41b006ed60 2024-04-25T01:23:00.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.191+0000 7f41af7fe640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f419006be30 0x7f419006e2f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f41a0006fd0 tx=0x7f41a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.195+0000 7f41acff9640 1 -- 172.21.15.77:0/542224044 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4198058d50 con 0x7f41b006ed60 2024-04-25T01:23:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.369+0000 7f41b68cd640 1 -- 172.21.15.77:0/542224044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41b0074ce0 con 0x7f41b006ed60 2024-04-25T01:23:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.370+0000 7f41acff9640 1 -- 172.21.15.77:0/542224044 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16942 (secure 0 0 0) 0x7f4198058650 con 0x7f41b006ed60 2024-04-25T01:23:00.350 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:00.351 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":38,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:58.725224+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.457887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.029883+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:00.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 -- 172.21.15.77:0/542224044 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f419006be30 msgr2=0x7f419006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f419006be30 0x7f419006e2f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f41a0006fd0 tx=0x7f41a0008040 comp rx=0 tx=0).stop 2024-04-25T01:23:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 -- 172.21.15.77:0/542224044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41b006ed60 msgr2=0x7f41b00acc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41b006ed60 0x7f41b00acc20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f419800c400 tx=0x7f419800c8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 -- 172.21.15.77:0/542224044 shutdown_connections 2024-04-25T01:23:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f419006be30 0x7f419006e2f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b0143d20 0x7f41b00c5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41b0143940 0x7f41b015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 --2- 172.21.15.77:0/542224044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41b006ed60 0x7f41b00acc20 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 -- 172.21.15.77:0/542224044 >> 172.21.15.77:0/542224044 conn(0x7f41b00b1820 msgr2=0x7f41b00cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 -- 172.21.15.77:0/542224044 shutdown_connections 2024-04-25T01:23:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.372+0000 7f41b68cd640 1 -- 172.21.15.77:0/542224044 wait complete. 2024-04-25T01:23:00.364 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T01:23:00.364 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:23:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.479+0000 7ff2c74fc640 1 Processor -- start 2024-04-25T01:23:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.480+0000 7ff2c74fc640 1 -- start start 2024-04-25T01:23:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.480+0000 7ff2c74fc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff2c0143d20 0x7ff2c015ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.480+0000 7ff2c74fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c006ed60 0x7ff2c015f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.480+0000 7ff2c74fc640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2c015f770 con 0x7ff2c0143940 2024-04-25T01:23:00.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.480+0000 7ff2c74fc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff2c0070e60 con 0x7ff2c0143d20 2024-04-25T01:23:00.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.480+0000 7ff2c74fc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff2c0071140 con 0x7ff2c006ed60 2024-04-25T01:23:00.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.481+0000 7ff2c4a97640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff2c0143d20 0x7ff2c015ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.481+0000 7ff2c5a99640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c006ed60 0x7ff2c015f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.481+0000 7ff2c4a97640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff2c0143d20 0x7ff2c015ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:35294/0 (socket says 172.21.15.77:35294) 2024-04-25T01:23:00.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.481+0000 7ff2c5a99640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c006ed60 0x7ff2c015f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:53308/0 (socket says 172.21.15.77:53308) 2024-04-25T01:23:00.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.481+0000 7ff2c4a97640 1 -- 172.21.15.77:0/2961248374 learned_addr learned my addr 172.21.15.77:0/2961248374 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:00.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.481+0000 7ff2c5298640 1 --1- 172.21.15.77:0/2961248374 >> v1:172.21.15.121:6789/0 conn(0x7ff2c0143940 0x7ff2c006e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:59578/0 (socket says 172.21.15.77:59578) 2024-04-25T01:23:00.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.481+0000 7ff2c5a99640 1 -- 172.21.15.77:0/2961248374 >> v2:172.21.15.121:3300/0 conn(0x7ff2c0143d20 msgr2=0x7ff2c015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:00.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.481+0000 7ff2c5a99640 1 --2- 172.21.15.77:0/2961248374 >> v2:172.21.15.121:3300/0 conn(0x7ff2c0143d20 0x7ff2c015ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.481+0000 7ff2c5a99640 1 -- 172.21.15.77:0/2961248374 >> v1:172.21.15.121:6789/0 conn(0x7ff2c0143940 legacy=0x7ff2c006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.481+0000 7ff2c5a99640 1 -- 172.21.15.77:0/2961248374 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c0071420 con 0x7ff2c006ed60 2024-04-25T01:23:00.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.481+0000 7ff2c4a97640 1 --2- 172.21.15.77:0/2961248374 >> v2:172.21.15.121:3300/0 conn(0x7ff2c0143d20 0x7ff2c015ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:00.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.482+0000 7ff2c5a99640 1 --2- 172.21.15.77:0/2961248374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c006ed60 0x7ff2c015f230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8009dc0 tx=0x7ff2b802a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0df7250781a05fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.482+0000 7ff2b7fff640 1 -- 172.21.15.77:0/2961248374 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b802ae60 con 0x7ff2c006ed60 2024-04-25T01:23:00.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.482+0000 7ff2b7fff640 1 -- 172.21.15.77:0/2961248374 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2b8007740 con 0x7ff2c006ed60 2024-04-25T01:23:00.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.482+0000 7ff2b7fff640 1 -- 172.21.15.77:0/2961248374 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8007b10 con 0x7ff2c006ed60 2024-04-25T01:23:00.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.482+0000 7ff2c74fc640 1 -- 172.21.15.77:0/2961248374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c006ed60 msgr2=0x7ff2c015f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.482+0000 7ff2c74fc640 1 --2- 172.21.15.77:0/2961248374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c006ed60 0x7ff2c015f230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8009dc0 tx=0x7ff2b802a680 comp rx=0 tx=0).stop 2024-04-25T01:23:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.483+0000 7ff2c74fc640 1 -- 172.21.15.77:0/2961248374 shutdown_connections 2024-04-25T01:23:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.483+0000 7ff2c74fc640 1 --2- 172.21.15.77:0/2961248374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c006ed60 0x7ff2c015f230 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.483+0000 7ff2c74fc640 1 --2- 172.21.15.77:0/2961248374 >> v2:172.21.15.121:3300/0 conn(0x7ff2c0143d20 0x7ff2c015ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.483+0000 7ff2c74fc640 1 -- 172.21.15.77:0/2961248374 >> 172.21.15.77:0/2961248374 conn(0x7ff2c00b1820 msgr2=0x7ff2c00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.483+0000 7ff2c74fc640 1 -- 172.21.15.77:0/2961248374 shutdown_connections 2024-04-25T01:23:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.483+0000 7ff2c74fc640 1 -- 172.21.15.77:0/2961248374 wait complete. 2024-04-25T01:23:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.484+0000 7ff2c74fc640 1 Processor -- start 2024-04-25T01:23:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.484+0000 7ff2c74fc640 1 -- start start 2024-04-25T01:23:00.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.484+0000 7ff2c74fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2c006ed60 0x7ff2c01f5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.484+0000 7ff2c74fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c0143940 0x7ff2c00b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.484+0000 7ff2c74fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c0143d20 0x7ff2c01540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.484+0000 7ff2c74fc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c0071da0 con 0x7ff2c0143940 2024-04-25T01:23:00.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.484+0000 7ff2c74fc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c0071c20 con 0x7ff2c006ed60 2024-04-25T01:23:00.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c74fc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff2c0071f20 con 0x7ff2c0143d20 2024-04-25T01:23:00.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c4a97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c0143940 0x7ff2c00b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c5298640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2c006ed60 0x7ff2c01f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c4a97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c0143940 0x7ff2c00b0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44106/0 (socket says 172.21.15.77:44106) 2024-04-25T01:23:00.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c4a97640 1 -- 172.21.15.77:0/1483794545 learned_addr learned my addr 172.21.15.77:0/1483794545 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:00.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c5a99640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c0143d20 0x7ff2c01540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c4a97640 1 -- 172.21.15.77:0/1483794545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c0143d20 msgr2=0x7ff2c01540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c4a97640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c0143d20 0x7ff2c01540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c4a97640 1 -- 172.21.15.77:0/1483794545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2c006ed60 msgr2=0x7ff2c01f5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c4a97640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2c006ed60 0x7ff2c01f5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c4a97640 1 -- 172.21.15.77:0/1483794545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c01f1a30 con 0x7ff2c0143940 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c4a97640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c0143940 0x7ff2c00b0c40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc00e9b0 tx=0x7ff2bc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2b5ffb640 1 -- 172.21.15.77:0/1483794545 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc017070 con 0x7ff2c0143940 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2b5ffb640 1 -- 172.21.15.77:0/1483794545 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2bc00c9a0 con 0x7ff2c0143940 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2c74fc640 1 -- 172.21.15.77:0/1483794545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2c01f1d10 con 0x7ff2c0143940 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.485+0000 7ff2b5ffb640 1 -- 172.21.15.77:0/1483794545 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc00cd50 con 0x7ff2c0143940 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.486+0000 7ff2c74fc640 1 -- 172.21.15.77:0/1483794545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2c01f22d0 con 0x7ff2c0143940 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.486+0000 7ff2c74fc640 1 -- 172.21.15.77:0/1483794545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2c0068fb0 con 0x7ff2c0143940 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.487+0000 7ff2b5ffb640 1 -- 172.21.15.77:0/1483794545 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff2bc01a070 con 0x7ff2c0143940 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.487+0000 7ff2b5ffb640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2a406be30 0x7ff2a406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.487+0000 7ff2b5ffb640 1 -- 172.21.15.77:0/1483794545 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7ff2bc08b190 con 0x7ff2c0143940 2024-04-25T01:23:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.487+0000 7ff2c5298640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2a406be30 0x7ff2a406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.488+0000 7ff2c5298640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2a406be30 0x7ff2a406e2f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff2c015d170 tx=0x7ff2b0007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.492+0000 7ff2b5ffb640 1 -- 172.21.15.77:0/1483794545 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff2bc059360 con 0x7ff2c0143940 2024-04-25T01:23:00.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.661+0000 7ff2c74fc640 1 -- 172.21.15.77:0/1483794545 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff2c006d150 con 0x7ff2a406be30 2024-04-25T01:23:00.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.667+0000 7ff2b5ffb640 1 -- 172.21.15.77:0/1483794545 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251889 (secure 0 0 0) 0x7ff2c006d150 con 0x7ff2a406be30 2024-04-25T01:23:00.650 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:00.651 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:23:00.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 -- 172.21.15.77:0/1483794545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2a406be30 msgr2=0x7ff2a406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2a406be30 0x7ff2a406e2f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff2c015d170 tx=0x7ff2b0007e80 comp rx=0 tx=0).stop 2024-04-25T01:23:00.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 -- 172.21.15.77:0/1483794545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c0143940 msgr2=0x7ff2c00b0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c0143940 0x7ff2c00b0c40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc00e9b0 tx=0x7ff2bc00ee70 comp rx=0 tx=0).stop 2024-04-25T01:23:00.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 -- 172.21.15.77:0/1483794545 shutdown_connections 2024-04-25T01:23:00.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2a406be30 0x7ff2a406e2f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c0143940 0x7ff2c00b0c40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2c0143d20 0x7ff2c01540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 --2- 172.21.15.77:0/1483794545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2c006ed60 0x7ff2c01f5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 -- 172.21.15.77:0/1483794545 >> 172.21.15.77:0/1483794545 conn(0x7ff2c00b1820 msgr2=0x7ff2c00cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:00.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 -- 172.21.15.77:0/1483794545 shutdown_connections 2024-04-25T01:23:00.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.670+0000 7ff2c74fc640 1 -- 172.21.15.77:0/1483794545 wait complete. 2024-04-25T01:23:00.661 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-04-25T01:22:59.171084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":110,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220592,"kb_used_data":4408,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754128,"statfs":{"total":773094113280,"available":772868227072,"internally_reserved":0,"allocated":4513792,"data_stored":2597601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000551"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.737687+0000","last_change":"2024-04-25T01:22:57.735260+0000","last_active":"2024-04-25T01:22:57.737687+0000","last_peered":"2024-04-25T01:22:57.737687+0000","last_clean":"2024-04-25T01:22:57.737687+0000","last_became_active":"2024-04-25T01:22:57.735009+0000","last_became_peered":"2024-04-25T01:22:57.735009+0000","last_unstale":"2024-04-25T01:22:57.737687+0000","last_undegraded":"2024-04-25T01:22:57.737687+0000","last_fullsized":"2024-04-25T01:22:57.737687+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:51.207286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733471+0000","last_change":"2024-04-25T01:22:57.727933+0000","last_active":"2024-04-25T01:22:57.733471+0000","last_peered":"2024-04-25T01:22:57.733471+0000","last_clean":"2024-04-25T01:22:57.733471+0000","last_became_active":"2024-04-25T01:22:57.727644+0000","last_became_peered":"2024-04-25T01:22:57.727644+0000","last_unstale":"2024-04-25T01:22:57.733471+0000","last_undegraded":"2024-04-25T01:22:57.733471+0000","last_fullsized":"2024-04-25T01:22:57.733471+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:21.098676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733784+0000","last_change":"2024-04-25T01:22:57.727233+0000","last_active":"2024-04-25T01:22:57.733784+0000","last_peered":"2024-04-25T01:22:57.733784+0000","last_clean":"2024-04-25T01:22:57.733784+0000","last_became_active":"2024-04-25T01:22:57.726916+0000","last_became_peered":"2024-04-25T01:22:57.726916+0000","last_unstale":"2024-04-25T01:22:57.733784+0000","last_undegraded":"2024-04-25T01:22:57.733784+0000","last_fullsized":"2024-04-25T01:22:57.733784+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:32.419309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732969+0000","last_change":"2024-04-25T01:22:57.726672+0000","last_active":"2024-04-25T01:22:57.732969+0000","last_peered":"2024-04-25T01:22:57.732969+0000","last_clean":"2024-04-25T01:22:57.732969+0000","last_became_active":"2024-04-25T01:22:57.726415+0000","last_became_peered":"2024-04-25T01:22:57.726415+0000","last_unstale":"2024-04-25T01:22:57.732969+0000","last_undegraded":"2024-04-25T01:22:57.732969+0000","last_fullsized":"2024-04-25T01:22:57.732969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:49.633304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.726291+0000","last_change":"2024-04-25T01:22:57.725249+0000","last_active":"2024-04-25T01:22:57.726291+0000","last_peered":"2024-04-25T01:22:57.726291+0000","last_clean":"2024-04-25T01:22:57.726291+0000","last_became_active":"2024-04-25T01:22:57.724966+0000","last_became_peered":"2024-04-25T01:22:57.724966+0000","last_unstale":"2024-04-25T01:22:57.726291+0000","last_undegraded":"2024-04-25T01:22:57.726291+0000","last_fullsized":"2024-04-25T01:22:57.726291+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:24:40.109414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747873+0000","last_change":"2024-04-25T01:22:57.746149+0000","last_active":"2024-04-25T01:22:57.747873+0000","last_peered":"2024-04-25T01:22:57.747873+0000","last_clean":"2024-04-25T01:22:57.747873+0000","last_became_active":"2024-04-25T01:22:57.745874+0000","last_became_peered":"2024-04-25T01:22:57.745874+0000","last_unstale":"2024-04-25T01:22:57.747873+0000","last_undegraded":"2024-04-25T01:22:57.747873+0000","last_fullsized":"2024-04-25T01:22:57.747873+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:31:46.853259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731759+0000","last_change":"2024-04-25T01:22:57.726616+0000","last_active":"2024-04-25T01:22:57.731759+0000","last_peered":"2024-04-25T01:22:57.731759+0000","last_clean":"2024-04-25T01:22:57.731759+0000","last_became_active":"2024-04-25T01:22:57.725546+0000","last_became_peered":"2024-04-25T01:22:57.725546+0000","last_unstale":"2024-04-25T01:22:57.731759+0000","last_undegraded":"2024-04-25T01:22:57.731759+0000","last_fullsized":"2024-04-25T01:22:57.731759+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:46.990910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733498+0000","last_change":"2024-04-25T01:22:57.726672+0000","last_active":"2024-04-25T01:22:57.733498+0000","last_peered":"2024-04-25T01:22:57.733498+0000","last_clean":"2024-04-25T01:22:57.733498+0000","last_became_active":"2024-04-25T01:22:57.726415+0000","last_became_peered":"2024-04-25T01:22:57.726415+0000","last_unstale":"2024-04-25T01:22:57.733498+0000","last_undegraded":"2024-04-25T01:22:57.733498+0000","last_fullsized":"2024-04-25T01:22:57.733498+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:11:15.975135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.736711+0000","last_change":"2024-04-25T01:22:57.735723+0000","last_active":"2024-04-25T01:22:57.736711+0000","last_peered":"2024-04-25T01:22:57.736711+0000","last_clean":"2024-04-25T01:22:57.736711+0000","last_became_active":"2024-04-25T01:22:57.735437+0000","last_became_peered":"2024-04-25T01:22:57.735437+0000","last_unstale":"2024-04-25T01:22:57.736711+0000","last_undegraded":"2024-04-25T01:22:57.736711+0000","last_fullsized":"2024-04-25T01:22:57.736711+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:04.656993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741896+0000","last_change":"2024-04-25T01:22:57.737250+0000","last_active":"2024-04-25T01:22:57.741896+0000","last_peered":"2024-04-25T01:22:57.741896+0000","last_clean":"2024-04-25T01:22:57.741896+0000","last_became_active":"2024-04-25T01:22:57.736548+0000","last_became_peered":"2024-04-25T01:22:57.736548+0000","last_unstale":"2024-04-25T01:22:57.741896+0000","last_undegraded":"2024-04-25T01:22:57.741896+0000","last_fullsized":"2024-04-25T01:22:57.741896+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:21.098676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731531+0000","last_change":"2024-04-25T01:22:57.725538+0000","last_active":"2024-04-25T01:22:57.731531+0000","last_peered":"2024-04-25T01:22:57.731531+0000","last_clean":"2024-04-25T01:22:57.731531+0000","last_became_active":"2024-04-25T01:22:57.725248+0000","last_became_peered":"2024-04-25T01:22:57.725248+0000","last_unstale":"2024-04-25T01:22:57.731531+0000","last_undegraded":"2024-04-25T01:22:57.731531+0000","last_fullsized":"2024-04-25T01:22:57.731531+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:27.750839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733745+0000","last_change":"2024-04-25T01:22:57.727759+0000","last_active":"2024-04-25T01:22:57.733745+0000","last_peered":"2024-04-25T01:22:57.733745+0000","last_clean":"2024-04-25T01:22:57.733745+0000","last_became_active":"2024-04-25T01:22:57.726914+0000","last_became_peered":"2024-04-25T01:22:57.726914+0000","last_unstale":"2024-04-25T01:22:57.733745+0000","last_undegraded":"2024-04-25T01:22:57.733745+0000","last_fullsized":"2024-04-25T01:22:57.733745+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:42:00.259068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746428+0000","last_change":"2024-04-25T01:22:57.745087+0000","last_active":"2024-04-25T01:22:57.746428+0000","last_peered":"2024-04-25T01:22:57.746428+0000","last_clean":"2024-04-25T01:22:57.746428+0000","last_became_active":"2024-04-25T01:22:57.744782+0000","last_became_peered":"2024-04-25T01:22:57.744782+0000","last_unstale":"2024-04-25T01:22:57.746428+0000","last_undegraded":"2024-04-25T01:22:57.746428+0000","last_fullsized":"2024-04-25T01:22:57.746428+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:36.109995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733798+0000","last_change":"2024-04-25T01:22:57.730713+0000","last_active":"2024-04-25T01:22:57.733798+0000","last_peered":"2024-04-25T01:22:57.733798+0000","last_clean":"2024-04-25T01:22:57.733798+0000","last_became_active":"2024-04-25T01:22:57.730447+0000","last_became_peered":"2024-04-25T01:22:57.730447+0000","last_unstale":"2024-04-25T01:22:57.733798+0000","last_undegraded":"2024-04-25T01:22:57.733798+0000","last_fullsized":"2024-04-25T01:22:57.733798+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:14.946270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731780+0000","last_change":"2024-04-25T01:22:57.726836+0000","last_active":"2024-04-25T01:22:57.731780+0000","last_peered":"2024-04-25T01:22:57.731780+0000","last_clean":"2024-04-25T01:22:57.731780+0000","last_became_active":"2024-04-25T01:22:57.726374+0000","last_became_peered":"2024-04-25T01:22:57.726374+0000","last_unstale":"2024-04-25T01:22:57.731780+0000","last_undegraded":"2024-04-25T01:22:57.731780+0000","last_fullsized":"2024-04-25T01:22:57.731780+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:20:31.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.742984+0000","last_change":"2024-04-25T01:22:57.738767+0000","last_active":"2024-04-25T01:22:57.742984+0000","last_peered":"2024-04-25T01:22:57.742984+0000","last_clean":"2024-04-25T01:22:57.742984+0000","last_became_active":"2024-04-25T01:22:57.738490+0000","last_became_peered":"2024-04-25T01:22:57.738490+0000","last_unstale":"2024-04-25T01:22:57.742984+0000","last_undegraded":"2024-04-25T01:22:57.742984+0000","last_fullsized":"2024-04-25T01:22:57.742984+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:30.935419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744145+0000","last_change":"2024-04-25T01:22:57.739740+0000","last_active":"2024-04-25T01:22:57.744145+0000","last_peered":"2024-04-25T01:22:57.744145+0000","last_clean":"2024-04-25T01:22:57.744145+0000","last_became_active":"2024-04-25T01:22:57.739495+0000","last_became_peered":"2024-04-25T01:22:57.739495+0000","last_unstale":"2024-04-25T01:22:57.744145+0000","last_undegraded":"2024-04-25T01:22:57.744145+0000","last_fullsized":"2024-04-25T01:22:57.744145+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:46.990910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.735576+0000","last_change":"2024-04-25T01:22:57.734308+0000","last_active":"2024-04-25T01:22:57.735576+0000","last_peered":"2024-04-25T01:22:57.735576+0000","last_clean":"2024-04-25T01:22:57.735576+0000","last_became_active":"2024-04-25T01:22:57.733898+0000","last_became_peered":"2024-04-25T01:22:57.733898+0000","last_unstale":"2024-04-25T01:22:57.735576+0000","last_undegraded":"2024-04-25T01:22:57.735576+0000","last_fullsized":"2024-04-25T01:22:57.735576+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:27:21.003785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745092+0000","last_change":"2024-04-25T01:22:57.738449+0000","last_active":"2024-04-25T01:22:57.745092+0000","last_peered":"2024-04-25T01:22:57.745092+0000","last_clean":"2024-04-25T01:22:57.745092+0000","last_became_active":"2024-04-25T01:22:57.738173+0000","last_became_peered":"2024-04-25T01:22:57.738173+0000","last_unstale":"2024-04-25T01:22:57.745092+0000","last_undegraded":"2024-04-25T01:22:57.745092+0000","last_fullsized":"2024-04-25T01:22:57.745092+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:52.362227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.740198+0000","last_change":"2024-04-25T01:22:57.734387+0000","last_active":"2024-04-25T01:22:57.740198+0000","last_peered":"2024-04-25T01:22:57.740198+0000","last_clean":"2024-04-25T01:22:57.740198+0000","last_became_active":"2024-04-25T01:22:57.733813+0000","last_became_peered":"2024-04-25T01:22:57.733813+0000","last_unstale":"2024-04-25T01:22:57.740198+0000","last_undegraded":"2024-04-25T01:22:57.740198+0000","last_fullsized":"2024-04-25T01:22:57.740198+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:49.673258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.742122+0000","last_change":"2024-04-25T01:22:57.739077+0000","last_active":"2024-04-25T01:22:57.742122+0000","last_peered":"2024-04-25T01:22:57.742122+0000","last_clean":"2024-04-25T01:22:57.742122+0000","last_became_active":"2024-04-25T01:22:57.738793+0000","last_became_peered":"2024-04-25T01:22:57.738793+0000","last_unstale":"2024-04-25T01:22:57.742122+0000","last_undegraded":"2024-04-25T01:22:57.742122+0000","last_fullsized":"2024-04-25T01:22:57.742122+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:30:56.764099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733780+0000","last_change":"2024-04-25T01:22:57.728626+0000","last_active":"2024-04-25T01:22:57.733780+0000","last_peered":"2024-04-25T01:22:57.733780+0000","last_clean":"2024-04-25T01:22:57.733780+0000","last_became_active":"2024-04-25T01:22:57.728038+0000","last_became_peered":"2024-04-25T01:22:57.728038+0000","last_unstale":"2024-04-25T01:22:57.733780+0000","last_undegraded":"2024-04-25T01:22:57.733780+0000","last_fullsized":"2024-04-25T01:22:57.733780+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:42:00.259068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745333+0000","last_change":"2024-04-25T01:22:57.740087+0000","last_active":"2024-04-25T01:22:57.745333+0000","last_peered":"2024-04-25T01:22:57.745333+0000","last_clean":"2024-04-25T01:22:57.745333+0000","last_became_active":"2024-04-25T01:22:57.739806+0000","last_became_peered":"2024-04-25T01:22:57.739806+0000","last_unstale":"2024-04-25T01:22:57.745333+0000","last_undegraded":"2024-04-25T01:22:57.745333+0000","last_fullsized":"2024-04-25T01:22:57.745333+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:31:46.853259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.743561+0000","last_change":"2024-04-25T01:22:57.737147+0000","last_active":"2024-04-25T01:22:57.743561+0000","last_peered":"2024-04-25T01:22:57.743561+0000","last_clean":"2024-04-25T01:22:57.743561+0000","last_became_active":"2024-04-25T01:22:57.736833+0000","last_became_peered":"2024-04-25T01:22:57.736833+0000","last_unstale":"2024-04-25T01:22:57.743561+0000","last_undegraded":"2024-04-25T01:22:57.743561+0000","last_fullsized":"2024-04-25T01:22:57.743561+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:48.692052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744437+0000","last_change":"2024-04-25T01:22:57.737808+0000","last_active":"2024-04-25T01:22:57.744437+0000","last_peered":"2024-04-25T01:22:57.744437+0000","last_clean":"2024-04-25T01:22:57.744437+0000","last_became_active":"2024-04-25T01:22:57.737542+0000","last_became_peered":"2024-04-25T01:22:57.737542+0000","last_unstale":"2024-04-25T01:22:57.744437+0000","last_undegraded":"2024-04-25T01:22:57.744437+0000","last_fullsized":"2024-04-25T01:22:57.744437+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:20.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747275+0000","last_change":"2024-04-25T01:22:57.740095+0000","last_active":"2024-04-25T01:22:57.747275+0000","last_peered":"2024-04-25T01:22:57.747275+0000","last_clean":"2024-04-25T01:22:57.747275+0000","last_became_active":"2024-04-25T01:22:57.738683+0000","last_became_peered":"2024-04-25T01:22:57.738683+0000","last_unstale":"2024-04-25T01:22:57.747275+0000","last_undegraded":"2024-04-25T01:22:57.747275+0000","last_fullsized":"2024-04-25T01:22:57.747275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:27.750839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.740301+0000","last_change":"2024-04-25T01:22:57.736978+0000","last_active":"2024-04-25T01:22:57.740301+0000","last_peered":"2024-04-25T01:22:57.740301+0000","last_clean":"2024-04-25T01:22:57.740301+0000","last_became_active":"2024-04-25T01:22:57.736730+0000","last_became_peered":"2024-04-25T01:22:57.736730+0000","last_unstale":"2024-04-25T01:22:57.740301+0000","last_undegraded":"2024-04-25T01:22:57.740301+0000","last_fullsized":"2024-04-25T01:22:57.740301+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:50.266703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732450+0000","last_change":"2024-04-25T01:22:57.728395+0000","last_active":"2024-04-25T01:22:57.732450+0000","last_peered":"2024-04-25T01:22:57.732450+0000","last_clean":"2024-04-25T01:22:57.732450+0000","last_became_active":"2024-04-25T01:22:57.728156+0000","last_became_peered":"2024-04-25T01:22:57.728156+0000","last_unstale":"2024-04-25T01:22:57.732450+0000","last_undegraded":"2024-04-25T01:22:57.732450+0000","last_fullsized":"2024-04-25T01:22:57.732450+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:31:46.853259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746297+0000","last_change":"2024-04-25T01:22:57.739505+0000","last_active":"2024-04-25T01:22:57.746297+0000","last_peered":"2024-04-25T01:22:57.746297+0000","last_clean":"2024-04-25T01:22:57.746297+0000","last_became_active":"2024-04-25T01:22:57.738947+0000","last_became_peered":"2024-04-25T01:22:57.738947+0000","last_unstale":"2024-04-25T01:22:57.746297+0000","last_undegraded":"2024-04-25T01:22:57.746297+0000","last_fullsized":"2024-04-25T01:22:57.746297+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:32.419309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746722+0000","last_change":"2024-04-25T01:22:57.742213+0000","last_active":"2024-04-25T01:22:57.746722+0000","last_peered":"2024-04-25T01:22:57.746722+0000","last_clean":"2024-04-25T01:22:57.746722+0000","last_became_active":"2024-04-25T01:22:57.741969+0000","last_became_peered":"2024-04-25T01:22:57.741969+0000","last_unstale":"2024-04-25T01:22:57.746722+0000","last_undegraded":"2024-04-25T01:22:57.746722+0000","last_fullsized":"2024-04-25T01:22:57.746722+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:35.348011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745530+0000","last_change":"2024-04-25T01:22:57.741029+0000","last_active":"2024-04-25T01:22:57.745530+0000","last_peered":"2024-04-25T01:22:57.745530+0000","last_clean":"2024-04-25T01:22:57.745530+0000","last_became_active":"2024-04-25T01:22:57.739668+0000","last_became_peered":"2024-04-25T01:22:57.739668+0000","last_unstale":"2024-04-25T01:22:57.745530+0000","last_undegraded":"2024-04-25T01:22:57.745530+0000","last_fullsized":"2024-04-25T01:22:57.745530+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:12.932562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747402+0000","last_change":"2024-04-25T01:22:57.739263+0000","last_active":"2024-04-25T01:22:57.747402+0000","last_peered":"2024-04-25T01:22:57.747402+0000","last_clean":"2024-04-25T01:22:57.747402+0000","last_became_active":"2024-04-25T01:22:57.738991+0000","last_became_peered":"2024-04-25T01:22:57.738991+0000","last_unstale":"2024-04-25T01:22:57.747402+0000","last_undegraded":"2024-04-25T01:22:57.747402+0000","last_fullsized":"2024-04-25T01:22:57.747402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:46.990910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745556+0000","last_change":"2024-04-25T01:22:57.741276+0000","last_active":"2024-04-25T01:22:57.745556+0000","last_peered":"2024-04-25T01:22:57.745556+0000","last_clean":"2024-04-25T01:22:57.745556+0000","last_became_active":"2024-04-25T01:22:57.740486+0000","last_became_peered":"2024-04-25T01:22:57.740486+0000","last_unstale":"2024-04-25T01:22:57.745556+0000","last_undegraded":"2024-04-25T01:22:57.745556+0000","last_fullsized":"2024-04-25T01:22:57.745556+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:24.178740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":16,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744370+0000","last_change":"2024-04-25T01:22:56.735132+0000","last_active":"2024-04-25T01:22:57.744370+0000","last_peered":"2024-04-25T01:22:57.744370+0000","last_clean":"2024-04-25T01:22:57.744370+0000","last_became_active":"2024-04-25T01:22:56.734852+0000","last_became_peered":"2024-04-25T01:22:56.734852+0000","last_unstale":"2024-04-25T01:22:57.744370+0000","last_undegraded":"2024-04-25T01:22:57.744370+0000","last_fullsized":"2024-04-25T01:22:57.744370+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:47.687853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747451+0000","last_change":"2024-04-25T01:22:57.740734+0000","last_active":"2024-04-25T01:22:57.747451+0000","last_peered":"2024-04-25T01:22:57.747451+0000","last_clean":"2024-04-25T01:22:57.747451+0000","last_became_active":"2024-04-25T01:22:57.740171+0000","last_became_peered":"2024-04-25T01:22:57.740171+0000","last_unstale":"2024-04-25T01:22:57.747451+0000","last_undegraded":"2024-04-25T01:22:57.747451+0000","last_fullsized":"2024-04-25T01:22:57.747451+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:20:31.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746422+0000","last_change":"2024-04-25T01:22:56.739818+0000","last_active":"2024-04-25T01:22:57.746422+0000","last_peered":"2024-04-25T01:22:57.746422+0000","last_clean":"2024-04-25T01:22:57.746422+0000","last_became_active":"2024-04-25T01:22:56.739548+0000","last_became_peered":"2024-04-25T01:22:56.739548+0000","last_unstale":"2024-04-25T01:22:57.746422+0000","last_undegraded":"2024-04-25T01:22:57.746422+0000","last_fullsized":"2024-04-25T01:22:57.746422+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:34.343811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.737940+0000","last_change":"2024-04-25T01:22:57.736461+0000","last_active":"2024-04-25T01:22:57.737940+0000","last_peered":"2024-04-25T01:22:57.737940+0000","last_clean":"2024-04-25T01:22:57.737940+0000","last_became_active":"2024-04-25T01:22:57.736158+0000","last_became_peered":"2024-04-25T01:22:57.736158+0000","last_unstale":"2024-04-25T01:22:57.737940+0000","last_undegraded":"2024-04-25T01:22:57.737940+0000","last_fullsized":"2024-04-25T01:22:57.737940+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:51:28.858521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744015+0000","last_change":"2024-04-25T01:22:56.731264+0000","last_active":"2024-04-25T01:22:57.744015+0000","last_peered":"2024-04-25T01:22:57.744015+0000","last_clean":"2024-04-25T01:22:57.744015+0000","last_became_active":"2024-04-25T01:22:56.731002+0000","last_became_peered":"2024-04-25T01:22:56.731002+0000","last_unstale":"2024-04-25T01:22:57.744015+0000","last_undegraded":"2024-04-25T01:22:57.744015+0000","last_fullsized":"2024-04-25T01:22:57.744015+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:38.801055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741138+0000","last_change":"2024-04-25T01:22:57.738334+0000","last_active":"2024-04-25T01:22:57.741138+0000","last_peered":"2024-04-25T01:22:57.741138+0000","last_clean":"2024-04-25T01:22:57.741138+0000","last_became_active":"2024-04-25T01:22:57.738039+0000","last_became_peered":"2024-04-25T01:22:57.738039+0000","last_unstale":"2024-04-25T01:22:57.741138+0000","last_undegraded":"2024-04-25T01:22:57.741138+0000","last_fullsized":"2024-04-25T01:22:57.741138+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:34.609175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.748311+0000","last_change":"2024-04-25T01:22:56.738881+0000","last_active":"2024-04-25T01:22:57.748311+0000","last_peered":"2024-04-25T01:22:57.748311+0000","last_clean":"2024-04-25T01:22:57.748311+0000","last_became_active":"2024-04-25T01:22:56.738540+0000","last_became_peered":"2024-04-25T01:22:56.738540+0000","last_unstale":"2024-04-25T01:22:57.748311+0000","last_undegraded":"2024-04-25T01:22:57.748311+0000","last_fullsized":"2024-04-25T01:22:57.748311+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:49.262504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744845+0000","last_change":"2024-04-25T01:22:57.743201+0000","last_active":"2024-04-25T01:22:57.744845+0000","last_peered":"2024-04-25T01:22:57.744845+0000","last_clean":"2024-04-25T01:22:57.744845+0000","last_became_active":"2024-04-25T01:22:57.742881+0000","last_became_peered":"2024-04-25T01:22:57.742881+0000","last_unstale":"2024-04-25T01:22:57.744845+0000","last_undegraded":"2024-04-25T01:22:57.744845+0000","last_fullsized":"2024-04-25T01:22:57.744845+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:39.805254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741768+0000","last_change":"2024-04-25T01:22:56.738039+0000","last_active":"2024-04-25T01:22:57.741768+0000","last_peered":"2024-04-25T01:22:57.741768+0000","last_clean":"2024-04-25T01:22:57.741768+0000","last_became_active":"2024-04-25T01:22:56.737756+0000","last_became_peered":"2024-04-25T01:22:56.737756+0000","last_unstale":"2024-04-25T01:22:57.741768+0000","last_undegraded":"2024-04-25T01:22:57.741768+0000","last_fullsized":"2024-04-25T01:22:57.741768+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:03.652794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741665+0000","last_change":"2024-04-25T01:22:57.735729+0000","last_active":"2024-04-25T01:22:57.741665+0000","last_peered":"2024-04-25T01:22:57.741665+0000","last_clean":"2024-04-25T01:22:57.741665+0000","last_became_active":"2024-04-25T01:22:57.735470+0000","last_became_peered":"2024-04-25T01:22:57.735470+0000","last_unstale":"2024-04-25T01:22:57.741665+0000","last_undegraded":"2024-04-25T01:22:57.741665+0000","last_fullsized":"2024-04-25T01:22:57.741665+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:11:15.975135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746882+0000","last_change":"2024-04-25T01:22:56.735323+0000","last_active":"2024-04-25T01:22:57.746882+0000","last_peered":"2024-04-25T01:22:57.746882+0000","last_clean":"2024-04-25T01:22:57.746882+0000","last_became_active":"2024-04-25T01:22:56.735057+0000","last_became_peered":"2024-04-25T01:22:56.735057+0000","last_unstale":"2024-04-25T01:22:57.746882+0000","last_undegraded":"2024-04-25T01:22:57.746882+0000","last_fullsized":"2024-04-25T01:22:57.746882+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:41:59.254869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732682+0000","last_change":"2024-04-25T01:22:57.726536+0000","last_active":"2024-04-25T01:22:57.732682+0000","last_peered":"2024-04-25T01:22:57.732682+0000","last_clean":"2024-04-25T01:22:57.732682+0000","last_became_active":"2024-04-25T01:22:57.726292+0000","last_became_peered":"2024-04-25T01:22:57.726292+0000","last_unstale":"2024-04-25T01:22:57.732682+0000","last_undegraded":"2024-04-25T01:22:57.732682+0000","last_fullsized":"2024-04-25T01:22:57.732682+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:20.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731287+0000","last_change":"2024-04-25T01:22:56.735197+0000","last_active":"2024-04-25T01:22:57.731287+0000","last_peered":"2024-04-25T01:22:57.731287+0000","last_clean":"2024-04-25T01:22:57.731287+0000","last_became_active":"2024-04-25T01:22:56.734941+0000","last_became_peered":"2024-04-25T01:22:56.734941+0000","last_unstale":"2024-04-25T01:22:57.731287+0000","last_undegraded":"2024-04-25T01:22:57.731287+0000","last_fullsized":"2024-04-25T01:22:57.731287+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:00.786299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746752+0000","last_change":"2024-04-25T01:22:57.745499+0000","last_active":"2024-04-25T01:22:57.746752+0000","last_peered":"2024-04-25T01:22:57.746752+0000","last_clean":"2024-04-25T01:22:57.746752+0000","last_became_active":"2024-04-25T01:22:57.744892+0000","last_became_peered":"2024-04-25T01:22:57.744892+0000","last_unstale":"2024-04-25T01:22:57.746752+0000","last_undegraded":"2024-04-25T01:22:57.746752+0000","last_fullsized":"2024-04-25T01:22:57.746752+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:29.035814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.722213+0000","last_change":"2024-04-25T01:22:56.735581+0000","last_active":"2024-04-25T01:22:57.722213+0000","last_peered":"2024-04-25T01:22:57.722213+0000","last_clean":"2024-04-25T01:22:57.722213+0000","last_became_active":"2024-04-25T01:22:56.734108+0000","last_became_peered":"2024-04-25T01:22:56.734108+0000","last_unstale":"2024-04-25T01:22:57.722213+0000","last_undegraded":"2024-04-25T01:22:57.722213+0000","last_fullsized":"2024-04-25T01:22:57.722213+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:48.669059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.728661+0000","last_change":"2024-04-25T01:22:57.726038+0000","last_active":"2024-04-25T01:22:57.728661+0000","last_peered":"2024-04-25T01:22:57.728661+0000","last_clean":"2024-04-25T01:22:57.728661+0000","last_became_active":"2024-04-25T01:22:57.725758+0000","last_became_peered":"2024-04-25T01:22:57.725758+0000","last_unstale":"2024-04-25T01:22:57.728661+0000","last_undegraded":"2024-04-25T01:22:57.728661+0000","last_fullsized":"2024-04-25T01:22:57.728661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:17.910821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.722710+0000","last_change":"2024-04-25T01:22:56.732364+0000","last_active":"2024-04-25T01:22:57.722710+0000","last_peered":"2024-04-25T01:22:57.722710+0000","last_clean":"2024-04-25T01:22:57.722710+0000","last_became_active":"2024-04-25T01:22:56.732010+0000","last_became_peered":"2024-04-25T01:22:56.732010+0000","last_unstale":"2024-04-25T01:22:57.722710+0000","last_undegraded":"2024-04-25T01:22:57.722710+0000","last_fullsized":"2024-04-25T01:22:57.722710+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:11.928363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.740266+0000","last_change":"2024-04-25T01:22:57.737726+0000","last_active":"2024-04-25T01:22:57.740266+0000","last_peered":"2024-04-25T01:22:57.740266+0000","last_clean":"2024-04-25T01:22:57.740266+0000","last_became_active":"2024-04-25T01:22:57.737489+0000","last_became_peered":"2024-04-25T01:22:57.737489+0000","last_unstale":"2024-04-25T01:22:57.740266+0000","last_undegraded":"2024-04-25T01:22:57.740266+0000","last_fullsized":"2024-04-25T01:22:57.740266+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:39.805254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.722900+0000","last_change":"2024-04-25T01:22:56.733745+0000","last_active":"2024-04-25T01:22:57.722900+0000","last_peered":"2024-04-25T01:22:57.722900+0000","last_clean":"2024-04-25T01:22:57.722900+0000","last_became_active":"2024-04-25T01:22:56.733473+0000","last_became_peered":"2024-04-25T01:22:56.733473+0000","last_unstale":"2024-04-25T01:22:57.722900+0000","last_undegraded":"2024-04-25T01:22:57.722900+0000","last_fullsized":"2024-04-25T01:22:57.722900+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:29.931219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746859+0000","last_change":"2024-04-25T01:22:57.744178+0000","last_active":"2024-04-25T01:22:57.746859+0000","last_peered":"2024-04-25T01:22:57.746859+0000","last_clean":"2024-04-25T01:22:57.746859+0000","last_became_active":"2024-04-25T01:22:57.743904+0000","last_became_peered":"2024-04-25T01:22:57.743904+0000","last_unstale":"2024-04-25T01:22:57.746859+0000","last_undegraded":"2024-04-25T01:22:57.746859+0000","last_fullsized":"2024-04-25T01:22:57.746859+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:33.441640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747792+0000","last_change":"2024-04-25T01:22:56.736427+0000","last_active":"2024-04-25T01:22:57.747792+0000","last_peered":"2024-04-25T01:22:57.747792+0000","last_clean":"2024-04-25T01:22:57.747792+0000","last_became_active":"2024-04-25T01:22:56.736172+0000","last_became_peered":"2024-04-25T01:22:56.736172+0000","last_unstale":"2024-04-25T01:22:57.747792+0000","last_undegraded":"2024-04-25T01:22:57.747792+0000","last_fullsized":"2024-04-25T01:22:57.747792+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:00.786299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733750+0000","last_change":"2024-04-25T01:22:57.728468+0000","last_active":"2024-04-25T01:22:57.733750+0000","last_peered":"2024-04-25T01:22:57.733750+0000","last_clean":"2024-04-25T01:22:57.733750+0000","last_became_active":"2024-04-25T01:22:57.728223+0000","last_became_peered":"2024-04-25T01:22:57.728223+0000","last_unstale":"2024-04-25T01:22:57.733750+0000","last_undegraded":"2024-04-25T01:22:57.733750+0000","last_fullsized":"2024-04-25T01:22:57.733750+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:35.348011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.749371+0000","last_change":"2024-04-25T01:22:56.735560+0000","last_active":"2024-04-25T01:22:57.749371+0000","last_peered":"2024-04-25T01:22:57.749371+0000","last_clean":"2024-04-25T01:22:57.749371+0000","last_became_active":"2024-04-25T01:22:56.735311+0000","last_became_peered":"2024-04-25T01:22:56.735311+0000","last_unstale":"2024-04-25T01:22:57.749371+0000","last_undegraded":"2024-04-25T01:22:57.749371+0000","last_fullsized":"2024-04-25T01:22:57.749371+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:20:30.812750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.734032+0000","last_change":"2024-04-25T01:22:57.728203+0000","last_active":"2024-04-25T01:22:57.734032+0000","last_peered":"2024-04-25T01:22:57.734032+0000","last_clean":"2024-04-25T01:22:57.734032+0000","last_became_active":"2024-04-25T01:22:57.727927+0000","last_became_peered":"2024-04-25T01:22:57.727927+0000","last_unstale":"2024-04-25T01:22:57.734032+0000","last_undegraded":"2024-04-25T01:22:57.734032+0000","last_fullsized":"2024-04-25T01:22:57.734032+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:51:28.858521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745825+0000","last_change":"2024-04-25T01:22:56.736642+0000","last_active":"2024-04-25T01:22:57.745825+0000","last_peered":"2024-04-25T01:22:57.745825+0000","last_clean":"2024-04-25T01:22:57.745825+0000","last_became_active":"2024-04-25T01:22:56.736373+0000","last_became_peered":"2024-04-25T01:22:56.736373+0000","last_unstale":"2024-04-25T01:22:57.745825+0000","last_undegraded":"2024-04-25T01:22:57.745825+0000","last_fullsized":"2024-04-25T01:22:57.745825+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:31:45.849059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.737656+0000","last_change":"2024-04-25T01:22:57.736069+0000","last_active":"2024-04-25T01:22:57.737656+0000","last_peered":"2024-04-25T01:22:57.737656+0000","last_clean":"2024-04-25T01:22:57.737656+0000","last_became_active":"2024-04-25T01:22:57.735818+0000","last_became_peered":"2024-04-25T01:22:57.735818+0000","last_unstale":"2024-04-25T01:22:57.737656+0000","last_undegraded":"2024-04-25T01:22:57.737656+0000","last_fullsized":"2024-04-25T01:22:57.737656+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:30:56.764099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732368+0000","last_change":"2024-04-25T01:22:56.736359+0000","last_active":"2024-04-25T01:22:57.732368+0000","last_peered":"2024-04-25T01:22:57.732368+0000","last_clean":"2024-04-25T01:22:57.732368+0000","last_became_active":"2024-04-25T01:22:56.736097+0000","last_became_peered":"2024-04-25T01:22:56.736097+0000","last_unstale":"2024-04-25T01:22:57.732368+0000","last_undegraded":"2024-04-25T01:22:57.732368+0000","last_fullsized":"2024-04-25T01:22:57.732368+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:03.652794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744521+0000","last_change":"2024-04-25T01:22:57.737267+0000","last_active":"2024-04-25T01:22:57.744521+0000","last_peered":"2024-04-25T01:22:57.744521+0000","last_clean":"2024-04-25T01:22:57.744521+0000","last_became_active":"2024-04-25T01:22:57.737013+0000","last_became_peered":"2024-04-25T01:22:57.737013+0000","last_unstale":"2024-04-25T01:22:57.744521+0000","last_undegraded":"2024-04-25T01:22:57.744521+0000","last_fullsized":"2024-04-25T01:22:57.744521+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:04.656993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.727422+0000","last_change":"2024-04-25T01:22:56.735773+0000","last_active":"2024-04-25T01:22:57.727422+0000","last_peered":"2024-04-25T01:22:57.727422+0000","last_clean":"2024-04-25T01:22:57.727422+0000","last_became_active":"2024-04-25T01:22:56.735097+0000","last_became_peered":"2024-04-25T01:22:56.735097+0000","last_unstale":"2024-04-25T01:22:57.727422+0000","last_undegraded":"2024-04-25T01:22:57.727422+0000","last_fullsized":"2024-04-25T01:22:57.727422+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:55.266953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741142+0000","last_change":"2024-04-25T01:22:57.738267+0000","last_active":"2024-04-25T01:22:57.741142+0000","last_peered":"2024-04-25T01:22:57.741142+0000","last_clean":"2024-04-25T01:22:57.741142+0000","last_became_active":"2024-04-25T01:22:57.737985+0000","last_became_peered":"2024-04-25T01:22:57.737985+0000","last_unstale":"2024-04-25T01:22:57.741142+0000","last_undegraded":"2024-04-25T01:22:57.741142+0000","last_fullsized":"2024-04-25T01:22:57.741142+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:36.109995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":20,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747612+0000","last_change":"2024-04-25T01:22:56.733144+0000","last_active":"2024-04-25T01:22:57.747612+0000","last_peered":"2024-04-25T01:22:57.747612+0000","last_clean":"2024-04-25T01:22:57.747612+0000","last_became_active":"2024-04-25T01:22:56.732860+0000","last_became_peered":"2024-04-25T01:22:56.732860+0000","last_unstale":"2024-04-25T01:22:57.747612+0000","last_undegraded":"2024-04-25T01:22:57.747612+0000","last_fullsized":"2024-04-25T01:22:57.747612+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:51:27.854321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.742537+0000","last_change":"2024-04-25T01:22:57.741480+0000","last_active":"2024-04-25T01:22:57.742537+0000","last_peered":"2024-04-25T01:22:57.742537+0000","last_clean":"2024-04-25T01:22:57.742537+0000","last_became_active":"2024-04-25T01:22:57.741221+0000","last_became_peered":"2024-04-25T01:22:57.741221+0000","last_unstale":"2024-04-25T01:22:57.742537+0000","last_undegraded":"2024-04-25T01:22:57.742537+0000","last_fullsized":"2024-04-25T01:22:57.742537+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:17.967394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.742621+0000","last_change":"2024-04-25T01:22:56.734424+0000","last_active":"2024-04-25T01:22:57.742621+0000","last_peered":"2024-04-25T01:22:57.742621+0000","last_clean":"2024-04-25T01:22:57.742621+0000","last_became_active":"2024-04-25T01:22:56.734110+0000","last_became_peered":"2024-04-25T01:22:56.734110+0000","last_unstale":"2024-04-25T01:22:57.742621+0000","last_undegraded":"2024-04-25T01:22:57.742621+0000","last_fullsized":"2024-04-25T01:22:57.742621+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:32.437440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741915+0000","last_change":"2024-04-25T01:22:57.738104+0000","last_active":"2024-04-25T01:22:57.741915+0000","last_peered":"2024-04-25T01:22:57.741915+0000","last_clean":"2024-04-25T01:22:57.741915+0000","last_became_active":"2024-04-25T01:22:57.737849+0000","last_became_peered":"2024-04-25T01:22:57.737849+0000","last_unstale":"2024-04-25T01:22:57.741915+0000","last_undegraded":"2024-04-25T01:22:57.741915+0000","last_fullsized":"2024-04-25T01:22:57.741915+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:51.207286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746874+0000","last_change":"2024-04-25T01:22:56.733102+0000","last_active":"2024-04-25T01:22:57.746874+0000","last_peered":"2024-04-25T01:22:57.746874+0000","last_clean":"2024-04-25T01:22:57.746874+0000","last_became_active":"2024-04-25T01:22:56.732817+0000","last_became_peered":"2024-04-25T01:22:56.732817+0000","last_unstale":"2024-04-25T01:22:57.746874+0000","last_undegraded":"2024-04-25T01:22:57.746874+0000","last_fullsized":"2024-04-25T01:22:57.746874+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:27:19.999586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.739883+0000","last_change":"2024-04-25T01:22:57.739111+0000","last_active":"2024-04-25T01:22:57.739883+0000","last_peered":"2024-04-25T01:22:57.739883+0000","last_clean":"2024-04-25T01:22:57.739883+0000","last_became_active":"2024-04-25T01:22:57.738736+0000","last_became_peered":"2024-04-25T01:22:57.738736+0000","last_unstale":"2024-04-25T01:22:57.739883+0000","last_undegraded":"2024-04-25T01:22:57.739883+0000","last_fullsized":"2024-04-25T01:22:57.739883+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:01.790499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746921+0000","last_change":"2024-04-25T01:22:56.732329+0000","last_active":"2024-04-25T01:22:57.746921+0000","last_peered":"2024-04-25T01:22:57.746921+0000","last_clean":"2024-04-25T01:22:57.746921+0000","last_became_active":"2024-04-25T01:22:56.732047+0000","last_became_peered":"2024-04-25T01:22:56.732047+0000","last_unstale":"2024-04-25T01:22:57.746921+0000","last_undegraded":"2024-04-25T01:22:57.746921+0000","last_fullsized":"2024-04-25T01:22:57.746921+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:10.714645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732925+0000","last_change":"2024-04-25T01:22:57.727455+0000","last_active":"2024-04-25T01:22:57.732925+0000","last_peered":"2024-04-25T01:22:57.732925+0000","last_clean":"2024-04-25T01:22:57.732925+0000","last_became_active":"2024-04-25T01:22:57.727177+0000","last_became_peered":"2024-04-25T01:22:57.727177+0000","last_unstale":"2024-04-25T01:22:57.732925+0000","last_undegraded":"2024-04-25T01:22:57.732925+0000","last_fullsized":"2024-04-25T01:22:57.732925+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:56.271152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731083+0000","last_change":"2024-04-25T01:22:56.735935+0000","last_active":"2024-04-25T01:22:57.731083+0000","last_peered":"2024-04-25T01:22:57.731083+0000","last_clean":"2024-04-25T01:22:57.731083+0000","last_became_active":"2024-04-25T01:22:56.735652+0000","last_became_peered":"2024-04-25T01:22:56.735652+0000","last_unstale":"2024-04-25T01:22:57.731083+0000","last_undegraded":"2024-04-25T01:22:57.731083+0000","last_fullsized":"2024-04-25T01:22:57.731083+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:51:27.854321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.743013+0000","last_change":"2024-04-25T01:22:57.739266+0000","last_active":"2024-04-25T01:22:57.743013+0000","last_peered":"2024-04-25T01:22:57.743013+0000","last_clean":"2024-04-25T01:22:57.743013+0000","last_became_active":"2024-04-25T01:22:57.739022+0000","last_became_peered":"2024-04-25T01:22:57.739022+0000","last_unstale":"2024-04-25T01:22:57.743013+0000","last_undegraded":"2024-04-25T01:22:57.743013+0000","last_fullsized":"2024-04-25T01:22:57.743013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:27.750839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.749579+0000","last_change":"2024-04-25T01:22:56.736970+0000","last_active":"2024-04-25T01:22:57.749579+0000","last_peered":"2024-04-25T01:22:57.749579+0000","last_clean":"2024-04-25T01:22:57.749579+0000","last_became_active":"2024-04-25T01:22:56.736705+0000","last_became_peered":"2024-04-25T01:22:56.736705+0000","last_unstale":"2024-04-25T01:22:57.749579+0000","last_undegraded":"2024-04-25T01:22:57.749579+0000","last_fullsized":"2024-04-25T01:22:57.749579+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:02:24.825202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746427+0000","last_change":"2024-04-25T01:22:57.744271+0000","last_active":"2024-04-25T01:22:57.746427+0000","last_peered":"2024-04-25T01:22:57.746427+0000","last_clean":"2024-04-25T01:22:57.746427+0000","last_became_active":"2024-04-25T01:22:57.743996+0000","last_became_peered":"2024-04-25T01:22:57.743996+0000","last_unstale":"2024-04-25T01:22:57.746427+0000","last_undegraded":"2024-04-25T01:22:57.746427+0000","last_fullsized":"2024-04-25T01:22:57.746427+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:30.935419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.718229+0000","last_change":"2024-04-25T01:22:56.729921+0000","last_active":"2024-04-25T01:22:57.718229+0000","last_peered":"2024-04-25T01:22:57.718229+0000","last_clean":"2024-04-25T01:22:57.718229+0000","last_became_active":"2024-04-25T01:22:56.729647+0000","last_became_peered":"2024-04-25T01:22:56.729647+0000","last_unstale":"2024-04-25T01:22:57.718229+0000","last_undegraded":"2024-04-25T01:22:57.718229+0000","last_fullsized":"2024-04-25T01:22:57.718229+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:22.731853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.736403+0000","last_change":"2024-04-25T01:22:57.734499+0000","last_active":"2024-04-25T01:22:57.736403+0000","last_peered":"2024-04-25T01:22:57.736403+0000","last_clean":"2024-04-25T01:22:57.736403+0000","last_became_active":"2024-04-25T01:22:57.734241+0000","last_became_peered":"2024-04-25T01:22:57.734241+0000","last_unstale":"2024-04-25T01:22:57.736403+0000","last_undegraded":"2024-04-25T01:22:57.736403+0000","last_fullsized":"2024-04-25T01:22:57.736403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:24.178740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.725188+0000","last_change":"2024-04-25T01:22:56.734733+0000","last_active":"2024-04-25T01:22:57.725188+0000","last_peered":"2024-04-25T01:22:57.725188+0000","last_clean":"2024-04-25T01:22:57.725188+0000","last_became_active":"2024-04-25T01:22:56.734427+0000","last_became_peered":"2024-04-25T01:22:56.734427+0000","last_unstale":"2024-04-25T01:22:57.725188+0000","last_undegraded":"2024-04-25T01:22:57.725188+0000","last_fullsized":"2024-04-25T01:22:57.725188+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:13.942071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747199+0000","last_change":"2024-04-25T01:22:57.745963+0000","last_active":"2024-04-25T01:22:57.747199+0000","last_peered":"2024-04-25T01:22:57.747199+0000","last_clean":"2024-04-25T01:22:57.747199+0000","last_became_active":"2024-04-25T01:22:57.745221+0000","last_became_peered":"2024-04-25T01:22:57.745221+0000","last_unstale":"2024-04-25T01:22:57.747199+0000","last_undegraded":"2024-04-25T01:22:57.747199+0000","last_fullsized":"2024-04-25T01:22:57.747199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:34.172822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.718155+0000","last_change":"2024-04-25T01:22:56.732992+0000","last_active":"2024-04-25T01:22:57.718155+0000","last_peered":"2024-04-25T01:22:57.718155+0000","last_clean":"2024-04-25T01:22:57.718155+0000","last_became_active":"2024-04-25T01:22:56.732735+0000","last_became_peered":"2024-04-25T01:22:56.732735+0000","last_unstale":"2024-04-25T01:22:57.718155+0000","last_undegraded":"2024-04-25T01:22:57.718155+0000","last_fullsized":"2024-04-25T01:22:57.718155+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:25:03.869420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747371+0000","last_change":"2024-04-25T01:22:57.740010+0000","last_active":"2024-04-25T01:22:57.747371+0000","last_peered":"2024-04-25T01:22:57.747371+0000","last_clean":"2024-04-25T01:22:57.747371+0000","last_became_active":"2024-04-25T01:22:57.739731+0000","last_became_peered":"2024-04-25T01:22:57.739731+0000","last_unstale":"2024-04-25T01:22:57.747371+0000","last_undegraded":"2024-04-25T01:22:57.747371+0000","last_fullsized":"2024-04-25T01:22:57.747371+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:30.935419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747866+0000","last_change":"2024-04-25T01:22:56.736716+0000","last_active":"2024-04-25T01:22:57.747866+0000","last_peered":"2024-04-25T01:22:57.747866+0000","last_clean":"2024-04-25T01:22:57.747866+0000","last_became_active":"2024-04-25T01:22:56.736126+0000","last_became_peered":"2024-04-25T01:22:56.736126+0000","last_unstale":"2024-04-25T01:22:57.747866+0000","last_undegraded":"2024-04-25T01:22:57.747866+0000","last_fullsized":"2024-04-25T01:22:57.747866+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:38.801055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.740287+0000","last_change":"2024-04-25T01:22:57.738193+0000","last_active":"2024-04-25T01:22:57.740287+0000","last_peered":"2024-04-25T01:22:57.740287+0000","last_clean":"2024-04-25T01:22:57.740287+0000","last_became_active":"2024-04-25T01:22:57.737942+0000","last_became_peered":"2024-04-25T01:22:57.737942+0000","last_unstale":"2024-04-25T01:22:57.740287+0000","last_undegraded":"2024-04-25T01:22:57.740287+0000","last_fullsized":"2024-04-25T01:22:57.740287+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:33.441640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.718493+0000","last_change":"2024-04-25T01:22:56.728741+0000","last_active":"2024-04-25T01:22:57.718493+0000","last_peered":"2024-04-25T01:22:57.718493+0000","last_clean":"2024-04-25T01:22:57.718493+0000","last_became_active":"2024-04-25T01:22:56.728466+0000","last_became_peered":"2024-04-25T01:22:56.728466+0000","last_unstale":"2024-04-25T01:22:57.718493+0000","last_undegraded":"2024-04-25T01:22:57.718493+0000","last_fullsized":"2024-04-25T01:22:57.718493+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:36.492290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731666+0000","last_change":"2024-04-25T01:22:57.725909+0000","last_active":"2024-04-25T01:22:57.731666+0000","last_peered":"2024-04-25T01:22:57.731666+0000","last_clean":"2024-04-25T01:22:57.731666+0000","last_became_active":"2024-04-25T01:22:57.725269+0000","last_became_peered":"2024-04-25T01:22:57.725269+0000","last_unstale":"2024-04-25T01:22:57.731666+0000","last_undegraded":"2024-04-25T01:22:57.731666+0000","last_fullsized":"2024-04-25T01:22:57.731666+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:24.178740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.726800+0000","last_change":"2024-04-25T01:22:56.734636+0000","last_active":"2024-04-25T01:22:57.726800+0000","last_peered":"2024-04-25T01:22:57.726800+0000","last_clean":"2024-04-25T01:22:57.726800+0000","last_became_active":"2024-04-25T01:22:56.734356+0000","last_became_peered":"2024-04-25T01:22:56.734356+0000","last_unstale":"2024-04-25T01:22:57.726800+0000","last_undegraded":"2024-04-25T01:22:57.726800+0000","last_fullsized":"2024-04-25T01:22:57.726800+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:19.967406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.734056+0000","last_change":"2024-04-25T01:22:57.728749+0000","last_active":"2024-04-25T01:22:57.734056+0000","last_peered":"2024-04-25T01:22:57.734056+0000","last_clean":"2024-04-25T01:22:57.734056+0000","last_became_active":"2024-04-25T01:22:57.728388+0000","last_became_peered":"2024-04-25T01:22:57.728388+0000","last_unstale":"2024-04-25T01:22:57.734056+0000","last_undegraded":"2024-04-25T01:22:57.734056+0000","last_fullsized":"2024-04-25T01:22:57.734056+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:01.790499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.728006+0000","last_change":"2024-04-25T01:22:56.735626+0000","last_active":"2024-04-25T01:22:57.728006+0000","last_peered":"2024-04-25T01:22:57.728006+0000","last_clean":"2024-04-25T01:22:57.728006+0000","last_became_active":"2024-04-25T01:22:56.735332+0000","last_became_peered":"2024-04-25T01:22:56.735332+0000","last_unstale":"2024-04-25T01:22:57.728006+0000","last_undegraded":"2024-04-25T01:22:57.728006+0000","last_fullsized":"2024-04-25T01:22:57.728006+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:13.942071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747207+0000","last_change":"2024-04-25T01:22:57.738083+0000","last_active":"2024-04-25T01:22:57.747207+0000","last_peered":"2024-04-25T01:22:57.747207+0000","last_clean":"2024-04-25T01:22:57.747207+0000","last_became_active":"2024-04-25T01:22:57.737782+0000","last_became_peered":"2024-04-25T01:22:57.737782+0000","last_unstale":"2024-04-25T01:22:57.747207+0000","last_undegraded":"2024-04-25T01:22:57.747207+0000","last_fullsized":"2024-04-25T01:22:57.747207+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:33.441640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.728512+0000","last_change":"2024-04-25T01:22:56.738400+0000","last_active":"2024-04-25T01:22:57.728512+0000","last_peered":"2024-04-25T01:22:57.728512+0000","last_clean":"2024-04-25T01:22:57.728512+0000","last_became_active":"2024-04-25T01:22:56.738076+0000","last_became_peered":"2024-04-25T01:22:56.738076+0000","last_unstale":"2024-04-25T01:22:57.728512+0000","last_undegraded":"2024-04-25T01:22:57.728512+0000","last_fullsized":"2024-04-25T01:22:57.728512+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:51.358028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732705+0000","last_change":"2024-04-25T01:22:57.727337+0000","last_active":"2024-04-25T01:22:57.732705+0000","last_peered":"2024-04-25T01:22:57.732705+0000","last_clean":"2024-04-25T01:22:57.732705+0000","last_became_active":"2024-04-25T01:22:57.727094+0000","last_became_peered":"2024-04-25T01:22:57.727094+0000","last_unstale":"2024-04-25T01:22:57.732705+0000","last_undegraded":"2024-04-25T01:22:57.732705+0000","last_fullsized":"2024-04-25T01:22:57.732705+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:32.419309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.721945+0000","last_change":"2024-04-25T01:22:56.738188+0000","last_active":"2024-04-25T01:22:57.721945+0000","last_peered":"2024-04-25T01:22:57.721945+0000","last_clean":"2024-04-25T01:22:57.721945+0000","last_became_active":"2024-04-25T01:22:56.737878+0000","last_became_peered":"2024-04-25T01:22:56.737878+0000","last_unstale":"2024-04-25T01:22:57.721945+0000","last_undegraded":"2024-04-25T01:22:57.721945+0000","last_fullsized":"2024-04-25T01:22:57.721945+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:51.358028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745241+0000","last_change":"2024-04-25T01:22:57.739929+0000","last_active":"2024-04-25T01:22:57.745241+0000","last_peered":"2024-04-25T01:22:57.745241+0000","last_clean":"2024-04-25T01:22:57.745241+0000","last_became_active":"2024-04-25T01:22:57.739681+0000","last_became_peered":"2024-04-25T01:22:57.739681+0000","last_unstale":"2024-04-25T01:22:57.745241+0000","last_undegraded":"2024-04-25T01:22:57.745241+0000","last_fullsized":"2024-04-25T01:22:57.745241+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:20:31.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.721367+0000","last_change":"2024-04-25T01:22:56.728643+0000","last_active":"2024-04-25T01:22:57.721367+0000","last_peered":"2024-04-25T01:22:57.721367+0000","last_clean":"2024-04-25T01:22:57.721367+0000","last_became_active":"2024-04-25T01:22:56.728358+0000","last_became_peered":"2024-04-25T01:22:56.728358+0000","last_unstale":"2024-04-25T01:22:57.721367+0000","last_undegraded":"2024-04-25T01:22:57.721367+0000","last_fullsized":"2024-04-25T01:22:57.721367+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:45.986711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.738399+0000","last_change":"2024-04-25T01:22:57.733292+0000","last_active":"2024-04-25T01:22:57.738399+0000","last_peered":"2024-04-25T01:22:57.738399+0000","last_clean":"2024-04-25T01:22:57.738399+0000","last_became_active":"2024-04-25T01:22:57.733019+0000","last_became_peered":"2024-04-25T01:22:57.733019+0000","last_unstale":"2024-04-25T01:22:57.738399+0000","last_undegraded":"2024-04-25T01:22:57.738399+0000","last_fullsized":"2024-04-25T01:22:57.738399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:12.932562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.729800+0000","last_change":"2024-04-25T01:22:56.734589+0000","last_active":"2024-04-25T01:22:57.729800+0000","last_peered":"2024-04-25T01:22:57.729800+0000","last_clean":"2024-04-25T01:22:57.729800+0000","last_became_active":"2024-04-25T01:22:56.734312+0000","last_became_peered":"2024-04-25T01:22:56.734312+0000","last_unstale":"2024-04-25T01:22:57.729800+0000","last_undegraded":"2024-04-25T01:22:57.729800+0000","last_fullsized":"2024-04-25T01:22:57.729800+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:00.786299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":200,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.726586+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:22:57.726586+0000","last_peered":"2024-04-25T01:22:57.726586+0000","last_clean":"2024-04-25T01:22:57.726586+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:22:57.726586+0000","last_undegraded":"2024-04-25T01:22:57.726586+0000","last_fullsized":"2024-04-25T01:22:57.726586+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":183458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343832,"statfs":{"total":96636764160,"available":96608083968,"internally_reserved":0,"allocated":1007616,"data_stored":766569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:23:00.665 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:23:00.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.818+0000 7f2c053d5640 1 Processor -- start 2024-04-25T01:23:00.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.818+0000 7f2c053d5640 1 -- start start 2024-04-25T01:23:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2c053d5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c00161730 0x7f2c000d1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2c053d5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2c000d0d70 0x7f2c000b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2c053d5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c000cfbc0 con 0x7f2c000cf650 2024-04-25T01:23:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2c053d5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f2c000cfea0 con 0x7f2c00161730 2024-04-25T01:23:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2c053d5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2c000b34f0 con 0x7f2c000d0d70 2024-04-25T01:23:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2bfeffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c00161730 0x7f2c000d1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2bfeffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c00161730 0x7f2c000d1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44116/0 (socket says 172.21.15.77:44116) 2024-04-25T01:23:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2bf7fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2c000cf650 0x7f2c000d0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58468/0 (socket says 172.21.15.77:58468) 2024-04-25T01:23:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2bfeffd640 1 -- 172.21.15.77:0/2688537926 learned_addr learned my addr 172.21.15.77:0/2688537926 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2bff7fe640 1 --2- 172.21.15.77:0/2688537926 >> v2:172.21.15.121:3300/0 conn(0x7f2c000d0d70 0x7f2c000b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2bfeffd640 1 -- 172.21.15.77:0/2688537926 >> v2:172.21.15.121:3300/0 conn(0x7f2c000d0d70 msgr2=0x7f2c000b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2bfeffd640 1 --2- 172.21.15.77:0/2688537926 >> v2:172.21.15.121:3300/0 conn(0x7f2c000d0d70 0x7f2c000b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2bfeffd640 1 -- 172.21.15.77:0/2688537926 >> v1:172.21.15.77:6789/0 conn(0x7f2c000cf650 legacy=0x7f2c000d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2bfe7fc640 1 -- 172.21.15.77:0/2688537926 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 491770272 0 0) 0x7f2c000cfbc0 con 0x7f2c000cf650 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.819+0000 7f2bfeffd640 1 -- 172.21.15.77:0/2688537926 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c000b37d0 con 0x7f2c00161730 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.820+0000 7f2bfeffd640 1 --2- 172.21.15.77:0/2688537926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c00161730 0x7f2c000d1200 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2c000ce510 tx=0x7f2bf002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b5f742ba243d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.820+0000 7f2bfe7fc640 1 -- 172.21.15.77:0/2688537926 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0037070 con 0x7f2c00161730 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.820+0000 7f2bfe7fc640 1 -- 172.21.15.77:0/2688537926 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bf0032440 con 0x7f2c00161730 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.820+0000 7f2bfe7fc640 1 -- 172.21.15.77:0/2688537926 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0032810 con 0x7f2c00161730 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.821+0000 7f2c053d5640 1 -- 172.21.15.77:0/2688537926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c00161730 msgr2=0x7f2c000d1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.821+0000 7f2c053d5640 1 --2- 172.21.15.77:0/2688537926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c00161730 0x7f2c000d1200 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2c000ce510 tx=0x7f2bf002a270 comp rx=0 tx=0).stop 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.821+0000 7f2c053d5640 1 -- 172.21.15.77:0/2688537926 shutdown_connections 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.821+0000 7f2c053d5640 1 --2- 172.21.15.77:0/2688537926 >> v2:172.21.15.121:3300/0 conn(0x7f2c000d0d70 0x7f2c000b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.821+0000 7f2c053d5640 1 --2- 172.21.15.77:0/2688537926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c00161730 0x7f2c000d1200 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.821+0000 7f2c053d5640 1 -- 172.21.15.77:0/2688537926 >> 172.21.15.77:0/2688537926 conn(0x7f2c000d3f30 msgr2=0x7f2c000d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.821+0000 7f2c053d5640 1 -- 172.21.15.77:0/2688537926 shutdown_connections 2024-04-25T01:23:00.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.821+0000 7f2c053d5640 1 -- 172.21.15.77:0/2688537926 wait complete. 2024-04-25T01:23:00.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.822+0000 7f2c053d5640 1 Processor -- start 2024-04-25T01:23:00.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.822+0000 7f2c053d5640 1 -- start start 2024-04-25T01:23:00.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2c053d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c000cf650 0x7f2c00153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2c053d5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c000d0d70 0x7f2c000b6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2c053d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c00161730 0x7f2c000b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2c053d5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c000d18c0 con 0x7f2c00161730 2024-04-25T01:23:00.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2c053d5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2c000d1740 con 0x7f2c000d0d70 2024-04-25T01:23:00.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2c053d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2c00161b10 con 0x7f2c000cf650 2024-04-25T01:23:00.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c00161730 0x7f2c000b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c00161730 0x7f2c000b7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44132/0 (socket says 172.21.15.77:44132) 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bff7fe640 1 -- 172.21.15.77:0/835950377 learned_addr learned my addr 172.21.15.77:0/835950377 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bf7fff640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c000d0d70 0x7f2c000b6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bfeffd640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c000cf650 0x7f2c00153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bff7fe640 1 -- 172.21.15.77:0/835950377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c000cf650 msgr2=0x7f2c00153fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bff7fe640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c000cf650 0x7f2c00153fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bff7fe640 1 -- 172.21.15.77:0/835950377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c000d0d70 msgr2=0x7f2c000b6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bff7fe640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c000d0d70 0x7f2c000b6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bff7fe640 1 -- 172.21.15.77:0/835950377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c000b1940 con 0x7f2c00161730 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bf7fff640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c000d0d70 0x7f2c000b6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bfeffd640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c000cf650 0x7f2c00153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.823+0000 7f2bff7fe640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c00161730 0x7f2c000b7060 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8007e60 tx=0x7f2bf800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.824+0000 7f2bf77fe640 1 -- 172.21.15.77:0/835950377 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8014620 con 0x7f2c00161730 2024-04-25T01:23:00.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.824+0000 7f2bf77fe640 1 -- 172.21.15.77:0/835950377 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bf8019070 con 0x7f2c00161730 2024-04-25T01:23:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.824+0000 7f2c053d5640 1 -- 172.21.15.77:0/835950377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c000b1c20 con 0x7f2c00161730 2024-04-25T01:23:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.824+0000 7f2bf77fe640 1 -- 172.21.15.77:0/835950377 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8014a00 con 0x7f2c00161730 2024-04-25T01:23:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.824+0000 7f2c053d5640 1 -- 172.21.15.77:0/835950377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c000b21e0 con 0x7f2c00161730 2024-04-25T01:23:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.825+0000 7f2c053d5640 1 -- 172.21.15.77:0/835950377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c00068fb0 con 0x7f2c00161730 2024-04-25T01:23:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.825+0000 7f2bf77fe640 1 -- 172.21.15.77:0/835950377 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2bf8014ba0 con 0x7f2c00161730 2024-04-25T01:23:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.825+0000 7f2bf77fe640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2be006be30 0x7f2be006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.825+0000 7f2bfeffd640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2be006be30 0x7f2be006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.825+0000 7f2bf77fe640 1 -- 172.21.15.77:0/835950377 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f2bf808b930 con 0x7f2c00161730 2024-04-25T01:23:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.826+0000 7f2bfeffd640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2be006be30 0x7f2be006e2f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0034000 tx=0x7f2bf002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:00.830+0000 7f2bf77fe640 1 -- 172.21.15.77:0/835950377 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2bf805a150 con 0x7f2c00161730 2024-04-25T01:23:00.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.001+0000 7f2c053d5640 1 -- 172.21.15.77:0/835950377 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2c001602e0 con 0x7f2be006be30 2024-04-25T01:23:00.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.007+0000 7f2bf77fe640 1 -- 172.21.15.77:0/835950377 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251889 (secure 0 0 0) 0x7f2c001602e0 con 0x7f2be006be30 2024-04-25T01:23:00.989 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:00.991 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:23:00.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 -- 172.21.15.77:0/835950377 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2be006be30 msgr2=0x7f2be006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2be006be30 0x7f2be006e2f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0034000 tx=0x7f2bf002e000 comp rx=0 tx=0).stop 2024-04-25T01:23:00.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 -- 172.21.15.77:0/835950377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c00161730 msgr2=0x7f2c000b7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c00161730 0x7f2c000b7060 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8007e60 tx=0x7f2bf800e4b0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 -- 172.21.15.77:0/835950377 shutdown_connections 2024-04-25T01:23:00.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2be006be30 0x7f2be006e2f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c00161730 0x7f2c000b7060 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c000d0d70 0x7f2c000b6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 --2- 172.21.15.77:0/835950377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c000cf650 0x7f2c00153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 -- 172.21.15.77:0/835950377 >> 172.21.15.77:0/835950377 conn(0x7f2c000d3f30 msgr2=0x7f2c000b3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:00.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 -- 172.21.15.77:0/835950377 shutdown_connections 2024-04-25T01:23:00.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.010+0000 7f2c053d5640 1 -- 172.21.15.77:0/835950377 wait complete. 2024-04-25T01:23:01.001 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-04-25T01:22:59.171084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":110,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220592,"kb_used_data":4408,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754128,"statfs":{"total":773094113280,"available":772868227072,"internally_reserved":0,"allocated":4513792,"data_stored":2597601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000551"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.737687+0000","last_change":"2024-04-25T01:22:57.735260+0000","last_active":"2024-04-25T01:22:57.737687+0000","last_peered":"2024-04-25T01:22:57.737687+0000","last_clean":"2024-04-25T01:22:57.737687+0000","last_became_active":"2024-04-25T01:22:57.735009+0000","last_became_peered":"2024-04-25T01:22:57.735009+0000","last_unstale":"2024-04-25T01:22:57.737687+0000","last_undegraded":"2024-04-25T01:22:57.737687+0000","last_fullsized":"2024-04-25T01:22:57.737687+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:51.207286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733471+0000","last_change":"2024-04-25T01:22:57.727933+0000","last_active":"2024-04-25T01:22:57.733471+0000","last_peered":"2024-04-25T01:22:57.733471+0000","last_clean":"2024-04-25T01:22:57.733471+0000","last_became_active":"2024-04-25T01:22:57.727644+0000","last_became_peered":"2024-04-25T01:22:57.727644+0000","last_unstale":"2024-04-25T01:22:57.733471+0000","last_undegraded":"2024-04-25T01:22:57.733471+0000","last_fullsized":"2024-04-25T01:22:57.733471+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:21.098676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733784+0000","last_change":"2024-04-25T01:22:57.727233+0000","last_active":"2024-04-25T01:22:57.733784+0000","last_peered":"2024-04-25T01:22:57.733784+0000","last_clean":"2024-04-25T01:22:57.733784+0000","last_became_active":"2024-04-25T01:22:57.726916+0000","last_became_peered":"2024-04-25T01:22:57.726916+0000","last_unstale":"2024-04-25T01:22:57.733784+0000","last_undegraded":"2024-04-25T01:22:57.733784+0000","last_fullsized":"2024-04-25T01:22:57.733784+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:32.419309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732969+0000","last_change":"2024-04-25T01:22:57.726672+0000","last_active":"2024-04-25T01:22:57.732969+0000","last_peered":"2024-04-25T01:22:57.732969+0000","last_clean":"2024-04-25T01:22:57.732969+0000","last_became_active":"2024-04-25T01:22:57.726415+0000","last_became_peered":"2024-04-25T01:22:57.726415+0000","last_unstale":"2024-04-25T01:22:57.732969+0000","last_undegraded":"2024-04-25T01:22:57.732969+0000","last_fullsized":"2024-04-25T01:22:57.732969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:49.633304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.726291+0000","last_change":"2024-04-25T01:22:57.725249+0000","last_active":"2024-04-25T01:22:57.726291+0000","last_peered":"2024-04-25T01:22:57.726291+0000","last_clean":"2024-04-25T01:22:57.726291+0000","last_became_active":"2024-04-25T01:22:57.724966+0000","last_became_peered":"2024-04-25T01:22:57.724966+0000","last_unstale":"2024-04-25T01:22:57.726291+0000","last_undegraded":"2024-04-25T01:22:57.726291+0000","last_fullsized":"2024-04-25T01:22:57.726291+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:24:40.109414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747873+0000","last_change":"2024-04-25T01:22:57.746149+0000","last_active":"2024-04-25T01:22:57.747873+0000","last_peered":"2024-04-25T01:22:57.747873+0000","last_clean":"2024-04-25T01:22:57.747873+0000","last_became_active":"2024-04-25T01:22:57.745874+0000","last_became_peered":"2024-04-25T01:22:57.745874+0000","last_unstale":"2024-04-25T01:22:57.747873+0000","last_undegraded":"2024-04-25T01:22:57.747873+0000","last_fullsized":"2024-04-25T01:22:57.747873+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:31:46.853259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731759+0000","last_change":"2024-04-25T01:22:57.726616+0000","last_active":"2024-04-25T01:22:57.731759+0000","last_peered":"2024-04-25T01:22:57.731759+0000","last_clean":"2024-04-25T01:22:57.731759+0000","last_became_active":"2024-04-25T01:22:57.725546+0000","last_became_peered":"2024-04-25T01:22:57.725546+0000","last_unstale":"2024-04-25T01:22:57.731759+0000","last_undegraded":"2024-04-25T01:22:57.731759+0000","last_fullsized":"2024-04-25T01:22:57.731759+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:46.990910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733498+0000","last_change":"2024-04-25T01:22:57.726672+0000","last_active":"2024-04-25T01:22:57.733498+0000","last_peered":"2024-04-25T01:22:57.733498+0000","last_clean":"2024-04-25T01:22:57.733498+0000","last_became_active":"2024-04-25T01:22:57.726415+0000","last_became_peered":"2024-04-25T01:22:57.726415+0000","last_unstale":"2024-04-25T01:22:57.733498+0000","last_undegraded":"2024-04-25T01:22:57.733498+0000","last_fullsized":"2024-04-25T01:22:57.733498+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:11:15.975135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.736711+0000","last_change":"2024-04-25T01:22:57.735723+0000","last_active":"2024-04-25T01:22:57.736711+0000","last_peered":"2024-04-25T01:22:57.736711+0000","last_clean":"2024-04-25T01:22:57.736711+0000","last_became_active":"2024-04-25T01:22:57.735437+0000","last_became_peered":"2024-04-25T01:22:57.735437+0000","last_unstale":"2024-04-25T01:22:57.736711+0000","last_undegraded":"2024-04-25T01:22:57.736711+0000","last_fullsized":"2024-04-25T01:22:57.736711+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:04.656993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741896+0000","last_change":"2024-04-25T01:22:57.737250+0000","last_active":"2024-04-25T01:22:57.741896+0000","last_peered":"2024-04-25T01:22:57.741896+0000","last_clean":"2024-04-25T01:22:57.741896+0000","last_became_active":"2024-04-25T01:22:57.736548+0000","last_became_peered":"2024-04-25T01:22:57.736548+0000","last_unstale":"2024-04-25T01:22:57.741896+0000","last_undegraded":"2024-04-25T01:22:57.741896+0000","last_fullsized":"2024-04-25T01:22:57.741896+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:21.098676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731531+0000","last_change":"2024-04-25T01:22:57.725538+0000","last_active":"2024-04-25T01:22:57.731531+0000","last_peered":"2024-04-25T01:22:57.731531+0000","last_clean":"2024-04-25T01:22:57.731531+0000","last_became_active":"2024-04-25T01:22:57.725248+0000","last_became_peered":"2024-04-25T01:22:57.725248+0000","last_unstale":"2024-04-25T01:22:57.731531+0000","last_undegraded":"2024-04-25T01:22:57.731531+0000","last_fullsized":"2024-04-25T01:22:57.731531+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:27.750839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733745+0000","last_change":"2024-04-25T01:22:57.727759+0000","last_active":"2024-04-25T01:22:57.733745+0000","last_peered":"2024-04-25T01:22:57.733745+0000","last_clean":"2024-04-25T01:22:57.733745+0000","last_became_active":"2024-04-25T01:22:57.726914+0000","last_became_peered":"2024-04-25T01:22:57.726914+0000","last_unstale":"2024-04-25T01:22:57.733745+0000","last_undegraded":"2024-04-25T01:22:57.733745+0000","last_fullsized":"2024-04-25T01:22:57.733745+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:42:00.259068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746428+0000","last_change":"2024-04-25T01:22:57.745087+0000","last_active":"2024-04-25T01:22:57.746428+0000","last_peered":"2024-04-25T01:22:57.746428+0000","last_clean":"2024-04-25T01:22:57.746428+0000","last_became_active":"2024-04-25T01:22:57.744782+0000","last_became_peered":"2024-04-25T01:22:57.744782+0000","last_unstale":"2024-04-25T01:22:57.746428+0000","last_undegraded":"2024-04-25T01:22:57.746428+0000","last_fullsized":"2024-04-25T01:22:57.746428+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:36.109995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733798+0000","last_change":"2024-04-25T01:22:57.730713+0000","last_active":"2024-04-25T01:22:57.733798+0000","last_peered":"2024-04-25T01:22:57.733798+0000","last_clean":"2024-04-25T01:22:57.733798+0000","last_became_active":"2024-04-25T01:22:57.730447+0000","last_became_peered":"2024-04-25T01:22:57.730447+0000","last_unstale":"2024-04-25T01:22:57.733798+0000","last_undegraded":"2024-04-25T01:22:57.733798+0000","last_fullsized":"2024-04-25T01:22:57.733798+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:14.946270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731780+0000","last_change":"2024-04-25T01:22:57.726836+0000","last_active":"2024-04-25T01:22:57.731780+0000","last_peered":"2024-04-25T01:22:57.731780+0000","last_clean":"2024-04-25T01:22:57.731780+0000","last_became_active":"2024-04-25T01:22:57.726374+0000","last_became_peered":"2024-04-25T01:22:57.726374+0000","last_unstale":"2024-04-25T01:22:57.731780+0000","last_undegraded":"2024-04-25T01:22:57.731780+0000","last_fullsized":"2024-04-25T01:22:57.731780+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:20:31.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.742984+0000","last_change":"2024-04-25T01:22:57.738767+0000","last_active":"2024-04-25T01:22:57.742984+0000","last_peered":"2024-04-25T01:22:57.742984+0000","last_clean":"2024-04-25T01:22:57.742984+0000","last_became_active":"2024-04-25T01:22:57.738490+0000","last_became_peered":"2024-04-25T01:22:57.738490+0000","last_unstale":"2024-04-25T01:22:57.742984+0000","last_undegraded":"2024-04-25T01:22:57.742984+0000","last_fullsized":"2024-04-25T01:22:57.742984+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:30.935419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744145+0000","last_change":"2024-04-25T01:22:57.739740+0000","last_active":"2024-04-25T01:22:57.744145+0000","last_peered":"2024-04-25T01:22:57.744145+0000","last_clean":"2024-04-25T01:22:57.744145+0000","last_became_active":"2024-04-25T01:22:57.739495+0000","last_became_peered":"2024-04-25T01:22:57.739495+0000","last_unstale":"2024-04-25T01:22:57.744145+0000","last_undegraded":"2024-04-25T01:22:57.744145+0000","last_fullsized":"2024-04-25T01:22:57.744145+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:46.990910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.735576+0000","last_change":"2024-04-25T01:22:57.734308+0000","last_active":"2024-04-25T01:22:57.735576+0000","last_peered":"2024-04-25T01:22:57.735576+0000","last_clean":"2024-04-25T01:22:57.735576+0000","last_became_active":"2024-04-25T01:22:57.733898+0000","last_became_peered":"2024-04-25T01:22:57.733898+0000","last_unstale":"2024-04-25T01:22:57.735576+0000","last_undegraded":"2024-04-25T01:22:57.735576+0000","last_fullsized":"2024-04-25T01:22:57.735576+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:27:21.003785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745092+0000","last_change":"2024-04-25T01:22:57.738449+0000","last_active":"2024-04-25T01:22:57.745092+0000","last_peered":"2024-04-25T01:22:57.745092+0000","last_clean":"2024-04-25T01:22:57.745092+0000","last_became_active":"2024-04-25T01:22:57.738173+0000","last_became_peered":"2024-04-25T01:22:57.738173+0000","last_unstale":"2024-04-25T01:22:57.745092+0000","last_undegraded":"2024-04-25T01:22:57.745092+0000","last_fullsized":"2024-04-25T01:22:57.745092+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:52.362227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.740198+0000","last_change":"2024-04-25T01:22:57.734387+0000","last_active":"2024-04-25T01:22:57.740198+0000","last_peered":"2024-04-25T01:22:57.740198+0000","last_clean":"2024-04-25T01:22:57.740198+0000","last_became_active":"2024-04-25T01:22:57.733813+0000","last_became_peered":"2024-04-25T01:22:57.733813+0000","last_unstale":"2024-04-25T01:22:57.740198+0000","last_undegraded":"2024-04-25T01:22:57.740198+0000","last_fullsized":"2024-04-25T01:22:57.740198+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:49.673258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.742122+0000","last_change":"2024-04-25T01:22:57.739077+0000","last_active":"2024-04-25T01:22:57.742122+0000","last_peered":"2024-04-25T01:22:57.742122+0000","last_clean":"2024-04-25T01:22:57.742122+0000","last_became_active":"2024-04-25T01:22:57.738793+0000","last_became_peered":"2024-04-25T01:22:57.738793+0000","last_unstale":"2024-04-25T01:22:57.742122+0000","last_undegraded":"2024-04-25T01:22:57.742122+0000","last_fullsized":"2024-04-25T01:22:57.742122+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:30:56.764099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733780+0000","last_change":"2024-04-25T01:22:57.728626+0000","last_active":"2024-04-25T01:22:57.733780+0000","last_peered":"2024-04-25T01:22:57.733780+0000","last_clean":"2024-04-25T01:22:57.733780+0000","last_became_active":"2024-04-25T01:22:57.728038+0000","last_became_peered":"2024-04-25T01:22:57.728038+0000","last_unstale":"2024-04-25T01:22:57.733780+0000","last_undegraded":"2024-04-25T01:22:57.733780+0000","last_fullsized":"2024-04-25T01:22:57.733780+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:42:00.259068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745333+0000","last_change":"2024-04-25T01:22:57.740087+0000","last_active":"2024-04-25T01:22:57.745333+0000","last_peered":"2024-04-25T01:22:57.745333+0000","last_clean":"2024-04-25T01:22:57.745333+0000","last_became_active":"2024-04-25T01:22:57.739806+0000","last_became_peered":"2024-04-25T01:22:57.739806+0000","last_unstale":"2024-04-25T01:22:57.745333+0000","last_undegraded":"2024-04-25T01:22:57.745333+0000","last_fullsized":"2024-04-25T01:22:57.745333+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:31:46.853259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.743561+0000","last_change":"2024-04-25T01:22:57.737147+0000","last_active":"2024-04-25T01:22:57.743561+0000","last_peered":"2024-04-25T01:22:57.743561+0000","last_clean":"2024-04-25T01:22:57.743561+0000","last_became_active":"2024-04-25T01:22:57.736833+0000","last_became_peered":"2024-04-25T01:22:57.736833+0000","last_unstale":"2024-04-25T01:22:57.743561+0000","last_undegraded":"2024-04-25T01:22:57.743561+0000","last_fullsized":"2024-04-25T01:22:57.743561+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:48.692052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744437+0000","last_change":"2024-04-25T01:22:57.737808+0000","last_active":"2024-04-25T01:22:57.744437+0000","last_peered":"2024-04-25T01:22:57.744437+0000","last_clean":"2024-04-25T01:22:57.744437+0000","last_became_active":"2024-04-25T01:22:57.737542+0000","last_became_peered":"2024-04-25T01:22:57.737542+0000","last_unstale":"2024-04-25T01:22:57.744437+0000","last_undegraded":"2024-04-25T01:22:57.744437+0000","last_fullsized":"2024-04-25T01:22:57.744437+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:20.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747275+0000","last_change":"2024-04-25T01:22:57.740095+0000","last_active":"2024-04-25T01:22:57.747275+0000","last_peered":"2024-04-25T01:22:57.747275+0000","last_clean":"2024-04-25T01:22:57.747275+0000","last_became_active":"2024-04-25T01:22:57.738683+0000","last_became_peered":"2024-04-25T01:22:57.738683+0000","last_unstale":"2024-04-25T01:22:57.747275+0000","last_undegraded":"2024-04-25T01:22:57.747275+0000","last_fullsized":"2024-04-25T01:22:57.747275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:27.750839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.740301+0000","last_change":"2024-04-25T01:22:57.736978+0000","last_active":"2024-04-25T01:22:57.740301+0000","last_peered":"2024-04-25T01:22:57.740301+0000","last_clean":"2024-04-25T01:22:57.740301+0000","last_became_active":"2024-04-25T01:22:57.736730+0000","last_became_peered":"2024-04-25T01:22:57.736730+0000","last_unstale":"2024-04-25T01:22:57.740301+0000","last_undegraded":"2024-04-25T01:22:57.740301+0000","last_fullsized":"2024-04-25T01:22:57.740301+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:50.266703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732450+0000","last_change":"2024-04-25T01:22:57.728395+0000","last_active":"2024-04-25T01:22:57.732450+0000","last_peered":"2024-04-25T01:22:57.732450+0000","last_clean":"2024-04-25T01:22:57.732450+0000","last_became_active":"2024-04-25T01:22:57.728156+0000","last_became_peered":"2024-04-25T01:22:57.728156+0000","last_unstale":"2024-04-25T01:22:57.732450+0000","last_undegraded":"2024-04-25T01:22:57.732450+0000","last_fullsized":"2024-04-25T01:22:57.732450+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:31:46.853259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746297+0000","last_change":"2024-04-25T01:22:57.739505+0000","last_active":"2024-04-25T01:22:57.746297+0000","last_peered":"2024-04-25T01:22:57.746297+0000","last_clean":"2024-04-25T01:22:57.746297+0000","last_became_active":"2024-04-25T01:22:57.738947+0000","last_became_peered":"2024-04-25T01:22:57.738947+0000","last_unstale":"2024-04-25T01:22:57.746297+0000","last_undegraded":"2024-04-25T01:22:57.746297+0000","last_fullsized":"2024-04-25T01:22:57.746297+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:32.419309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746722+0000","last_change":"2024-04-25T01:22:57.742213+0000","last_active":"2024-04-25T01:22:57.746722+0000","last_peered":"2024-04-25T01:22:57.746722+0000","last_clean":"2024-04-25T01:22:57.746722+0000","last_became_active":"2024-04-25T01:22:57.741969+0000","last_became_peered":"2024-04-25T01:22:57.741969+0000","last_unstale":"2024-04-25T01:22:57.746722+0000","last_undegraded":"2024-04-25T01:22:57.746722+0000","last_fullsized":"2024-04-25T01:22:57.746722+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:35.348011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745530+0000","last_change":"2024-04-25T01:22:57.741029+0000","last_active":"2024-04-25T01:22:57.745530+0000","last_peered":"2024-04-25T01:22:57.745530+0000","last_clean":"2024-04-25T01:22:57.745530+0000","last_became_active":"2024-04-25T01:22:57.739668+0000","last_became_peered":"2024-04-25T01:22:57.739668+0000","last_unstale":"2024-04-25T01:22:57.745530+0000","last_undegraded":"2024-04-25T01:22:57.745530+0000","last_fullsized":"2024-04-25T01:22:57.745530+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:12.932562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747402+0000","last_change":"2024-04-25T01:22:57.739263+0000","last_active":"2024-04-25T01:22:57.747402+0000","last_peered":"2024-04-25T01:22:57.747402+0000","last_clean":"2024-04-25T01:22:57.747402+0000","last_became_active":"2024-04-25T01:22:57.738991+0000","last_became_peered":"2024-04-25T01:22:57.738991+0000","last_unstale":"2024-04-25T01:22:57.747402+0000","last_undegraded":"2024-04-25T01:22:57.747402+0000","last_fullsized":"2024-04-25T01:22:57.747402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:46.990910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745556+0000","last_change":"2024-04-25T01:22:57.741276+0000","last_active":"2024-04-25T01:22:57.745556+0000","last_peered":"2024-04-25T01:22:57.745556+0000","last_clean":"2024-04-25T01:22:57.745556+0000","last_became_active":"2024-04-25T01:22:57.740486+0000","last_became_peered":"2024-04-25T01:22:57.740486+0000","last_unstale":"2024-04-25T01:22:57.745556+0000","last_undegraded":"2024-04-25T01:22:57.745556+0000","last_fullsized":"2024-04-25T01:22:57.745556+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:24.178740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":16,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744370+0000","last_change":"2024-04-25T01:22:56.735132+0000","last_active":"2024-04-25T01:22:57.744370+0000","last_peered":"2024-04-25T01:22:57.744370+0000","last_clean":"2024-04-25T01:22:57.744370+0000","last_became_active":"2024-04-25T01:22:56.734852+0000","last_became_peered":"2024-04-25T01:22:56.734852+0000","last_unstale":"2024-04-25T01:22:57.744370+0000","last_undegraded":"2024-04-25T01:22:57.744370+0000","last_fullsized":"2024-04-25T01:22:57.744370+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:47.687853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747451+0000","last_change":"2024-04-25T01:22:57.740734+0000","last_active":"2024-04-25T01:22:57.747451+0000","last_peered":"2024-04-25T01:22:57.747451+0000","last_clean":"2024-04-25T01:22:57.747451+0000","last_became_active":"2024-04-25T01:22:57.740171+0000","last_became_peered":"2024-04-25T01:22:57.740171+0000","last_unstale":"2024-04-25T01:22:57.747451+0000","last_undegraded":"2024-04-25T01:22:57.747451+0000","last_fullsized":"2024-04-25T01:22:57.747451+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:20:31.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746422+0000","last_change":"2024-04-25T01:22:56.739818+0000","last_active":"2024-04-25T01:22:57.746422+0000","last_peered":"2024-04-25T01:22:57.746422+0000","last_clean":"2024-04-25T01:22:57.746422+0000","last_became_active":"2024-04-25T01:22:56.739548+0000","last_became_peered":"2024-04-25T01:22:56.739548+0000","last_unstale":"2024-04-25T01:22:57.746422+0000","last_undegraded":"2024-04-25T01:22:57.746422+0000","last_fullsized":"2024-04-25T01:22:57.746422+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:34.343811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.737940+0000","last_change":"2024-04-25T01:22:57.736461+0000","last_active":"2024-04-25T01:22:57.737940+0000","last_peered":"2024-04-25T01:22:57.737940+0000","last_clean":"2024-04-25T01:22:57.737940+0000","last_became_active":"2024-04-25T01:22:57.736158+0000","last_became_peered":"2024-04-25T01:22:57.736158+0000","last_unstale":"2024-04-25T01:22:57.737940+0000","last_undegraded":"2024-04-25T01:22:57.737940+0000","last_fullsized":"2024-04-25T01:22:57.737940+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:51:28.858521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744015+0000","last_change":"2024-04-25T01:22:56.731264+0000","last_active":"2024-04-25T01:22:57.744015+0000","last_peered":"2024-04-25T01:22:57.744015+0000","last_clean":"2024-04-25T01:22:57.744015+0000","last_became_active":"2024-04-25T01:22:56.731002+0000","last_became_peered":"2024-04-25T01:22:56.731002+0000","last_unstale":"2024-04-25T01:22:57.744015+0000","last_undegraded":"2024-04-25T01:22:57.744015+0000","last_fullsized":"2024-04-25T01:22:57.744015+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:38.801055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741138+0000","last_change":"2024-04-25T01:22:57.738334+0000","last_active":"2024-04-25T01:22:57.741138+0000","last_peered":"2024-04-25T01:22:57.741138+0000","last_clean":"2024-04-25T01:22:57.741138+0000","last_became_active":"2024-04-25T01:22:57.738039+0000","last_became_peered":"2024-04-25T01:22:57.738039+0000","last_unstale":"2024-04-25T01:22:57.741138+0000","last_undegraded":"2024-04-25T01:22:57.741138+0000","last_fullsized":"2024-04-25T01:22:57.741138+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:34.609175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.748311+0000","last_change":"2024-04-25T01:22:56.738881+0000","last_active":"2024-04-25T01:22:57.748311+0000","last_peered":"2024-04-25T01:22:57.748311+0000","last_clean":"2024-04-25T01:22:57.748311+0000","last_became_active":"2024-04-25T01:22:56.738540+0000","last_became_peered":"2024-04-25T01:22:56.738540+0000","last_unstale":"2024-04-25T01:22:57.748311+0000","last_undegraded":"2024-04-25T01:22:57.748311+0000","last_fullsized":"2024-04-25T01:22:57.748311+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:49.262504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744845+0000","last_change":"2024-04-25T01:22:57.743201+0000","last_active":"2024-04-25T01:22:57.744845+0000","last_peered":"2024-04-25T01:22:57.744845+0000","last_clean":"2024-04-25T01:22:57.744845+0000","last_became_active":"2024-04-25T01:22:57.742881+0000","last_became_peered":"2024-04-25T01:22:57.742881+0000","last_unstale":"2024-04-25T01:22:57.744845+0000","last_undegraded":"2024-04-25T01:22:57.744845+0000","last_fullsized":"2024-04-25T01:22:57.744845+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:39.805254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741768+0000","last_change":"2024-04-25T01:22:56.738039+0000","last_active":"2024-04-25T01:22:57.741768+0000","last_peered":"2024-04-25T01:22:57.741768+0000","last_clean":"2024-04-25T01:22:57.741768+0000","last_became_active":"2024-04-25T01:22:56.737756+0000","last_became_peered":"2024-04-25T01:22:56.737756+0000","last_unstale":"2024-04-25T01:22:57.741768+0000","last_undegraded":"2024-04-25T01:22:57.741768+0000","last_fullsized":"2024-04-25T01:22:57.741768+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:03.652794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741665+0000","last_change":"2024-04-25T01:22:57.735729+0000","last_active":"2024-04-25T01:22:57.741665+0000","last_peered":"2024-04-25T01:22:57.741665+0000","last_clean":"2024-04-25T01:22:57.741665+0000","last_became_active":"2024-04-25T01:22:57.735470+0000","last_became_peered":"2024-04-25T01:22:57.735470+0000","last_unstale":"2024-04-25T01:22:57.741665+0000","last_undegraded":"2024-04-25T01:22:57.741665+0000","last_fullsized":"2024-04-25T01:22:57.741665+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:11:15.975135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746882+0000","last_change":"2024-04-25T01:22:56.735323+0000","last_active":"2024-04-25T01:22:57.746882+0000","last_peered":"2024-04-25T01:22:57.746882+0000","last_clean":"2024-04-25T01:22:57.746882+0000","last_became_active":"2024-04-25T01:22:56.735057+0000","last_became_peered":"2024-04-25T01:22:56.735057+0000","last_unstale":"2024-04-25T01:22:57.746882+0000","last_undegraded":"2024-04-25T01:22:57.746882+0000","last_fullsized":"2024-04-25T01:22:57.746882+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:41:59.254869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732682+0000","last_change":"2024-04-25T01:22:57.726536+0000","last_active":"2024-04-25T01:22:57.732682+0000","last_peered":"2024-04-25T01:22:57.732682+0000","last_clean":"2024-04-25T01:22:57.732682+0000","last_became_active":"2024-04-25T01:22:57.726292+0000","last_became_peered":"2024-04-25T01:22:57.726292+0000","last_unstale":"2024-04-25T01:22:57.732682+0000","last_undegraded":"2024-04-25T01:22:57.732682+0000","last_fullsized":"2024-04-25T01:22:57.732682+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:20.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731287+0000","last_change":"2024-04-25T01:22:56.735197+0000","last_active":"2024-04-25T01:22:57.731287+0000","last_peered":"2024-04-25T01:22:57.731287+0000","last_clean":"2024-04-25T01:22:57.731287+0000","last_became_active":"2024-04-25T01:22:56.734941+0000","last_became_peered":"2024-04-25T01:22:56.734941+0000","last_unstale":"2024-04-25T01:22:57.731287+0000","last_undegraded":"2024-04-25T01:22:57.731287+0000","last_fullsized":"2024-04-25T01:22:57.731287+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:00.786299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746752+0000","last_change":"2024-04-25T01:22:57.745499+0000","last_active":"2024-04-25T01:22:57.746752+0000","last_peered":"2024-04-25T01:22:57.746752+0000","last_clean":"2024-04-25T01:22:57.746752+0000","last_became_active":"2024-04-25T01:22:57.744892+0000","last_became_peered":"2024-04-25T01:22:57.744892+0000","last_unstale":"2024-04-25T01:22:57.746752+0000","last_undegraded":"2024-04-25T01:22:57.746752+0000","last_fullsized":"2024-04-25T01:22:57.746752+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:29.035814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.722213+0000","last_change":"2024-04-25T01:22:56.735581+0000","last_active":"2024-04-25T01:22:57.722213+0000","last_peered":"2024-04-25T01:22:57.722213+0000","last_clean":"2024-04-25T01:22:57.722213+0000","last_became_active":"2024-04-25T01:22:56.734108+0000","last_became_peered":"2024-04-25T01:22:56.734108+0000","last_unstale":"2024-04-25T01:22:57.722213+0000","last_undegraded":"2024-04-25T01:22:57.722213+0000","last_fullsized":"2024-04-25T01:22:57.722213+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:48.669059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.728661+0000","last_change":"2024-04-25T01:22:57.726038+0000","last_active":"2024-04-25T01:22:57.728661+0000","last_peered":"2024-04-25T01:22:57.728661+0000","last_clean":"2024-04-25T01:22:57.728661+0000","last_became_active":"2024-04-25T01:22:57.725758+0000","last_became_peered":"2024-04-25T01:22:57.725758+0000","last_unstale":"2024-04-25T01:22:57.728661+0000","last_undegraded":"2024-04-25T01:22:57.728661+0000","last_fullsized":"2024-04-25T01:22:57.728661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:17.910821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.722710+0000","last_change":"2024-04-25T01:22:56.732364+0000","last_active":"2024-04-25T01:22:57.722710+0000","last_peered":"2024-04-25T01:22:57.722710+0000","last_clean":"2024-04-25T01:22:57.722710+0000","last_became_active":"2024-04-25T01:22:56.732010+0000","last_became_peered":"2024-04-25T01:22:56.732010+0000","last_unstale":"2024-04-25T01:22:57.722710+0000","last_undegraded":"2024-04-25T01:22:57.722710+0000","last_fullsized":"2024-04-25T01:22:57.722710+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:11.928363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.740266+0000","last_change":"2024-04-25T01:22:57.737726+0000","last_active":"2024-04-25T01:22:57.740266+0000","last_peered":"2024-04-25T01:22:57.740266+0000","last_clean":"2024-04-25T01:22:57.740266+0000","last_became_active":"2024-04-25T01:22:57.737489+0000","last_became_peered":"2024-04-25T01:22:57.737489+0000","last_unstale":"2024-04-25T01:22:57.740266+0000","last_undegraded":"2024-04-25T01:22:57.740266+0000","last_fullsized":"2024-04-25T01:22:57.740266+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:39.805254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.722900+0000","last_change":"2024-04-25T01:22:56.733745+0000","last_active":"2024-04-25T01:22:57.722900+0000","last_peered":"2024-04-25T01:22:57.722900+0000","last_clean":"2024-04-25T01:22:57.722900+0000","last_became_active":"2024-04-25T01:22:56.733473+0000","last_became_peered":"2024-04-25T01:22:56.733473+0000","last_unstale":"2024-04-25T01:22:57.722900+0000","last_undegraded":"2024-04-25T01:22:57.722900+0000","last_fullsized":"2024-04-25T01:22:57.722900+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:29.931219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746859+0000","last_change":"2024-04-25T01:22:57.744178+0000","last_active":"2024-04-25T01:22:57.746859+0000","last_peered":"2024-04-25T01:22:57.746859+0000","last_clean":"2024-04-25T01:22:57.746859+0000","last_became_active":"2024-04-25T01:22:57.743904+0000","last_became_peered":"2024-04-25T01:22:57.743904+0000","last_unstale":"2024-04-25T01:22:57.746859+0000","last_undegraded":"2024-04-25T01:22:57.746859+0000","last_fullsized":"2024-04-25T01:22:57.746859+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:33.441640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747792+0000","last_change":"2024-04-25T01:22:56.736427+0000","last_active":"2024-04-25T01:22:57.747792+0000","last_peered":"2024-04-25T01:22:57.747792+0000","last_clean":"2024-04-25T01:22:57.747792+0000","last_became_active":"2024-04-25T01:22:56.736172+0000","last_became_peered":"2024-04-25T01:22:56.736172+0000","last_unstale":"2024-04-25T01:22:57.747792+0000","last_undegraded":"2024-04-25T01:22:57.747792+0000","last_fullsized":"2024-04-25T01:22:57.747792+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:00.786299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.733750+0000","last_change":"2024-04-25T01:22:57.728468+0000","last_active":"2024-04-25T01:22:57.733750+0000","last_peered":"2024-04-25T01:22:57.733750+0000","last_clean":"2024-04-25T01:22:57.733750+0000","last_became_active":"2024-04-25T01:22:57.728223+0000","last_became_peered":"2024-04-25T01:22:57.728223+0000","last_unstale":"2024-04-25T01:22:57.733750+0000","last_undegraded":"2024-04-25T01:22:57.733750+0000","last_fullsized":"2024-04-25T01:22:57.733750+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:35.348011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.749371+0000","last_change":"2024-04-25T01:22:56.735560+0000","last_active":"2024-04-25T01:22:57.749371+0000","last_peered":"2024-04-25T01:22:57.749371+0000","last_clean":"2024-04-25T01:22:57.749371+0000","last_became_active":"2024-04-25T01:22:56.735311+0000","last_became_peered":"2024-04-25T01:22:56.735311+0000","last_unstale":"2024-04-25T01:22:57.749371+0000","last_undegraded":"2024-04-25T01:22:57.749371+0000","last_fullsized":"2024-04-25T01:22:57.749371+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:20:30.812750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.734032+0000","last_change":"2024-04-25T01:22:57.728203+0000","last_active":"2024-04-25T01:22:57.734032+0000","last_peered":"2024-04-25T01:22:57.734032+0000","last_clean":"2024-04-25T01:22:57.734032+0000","last_became_active":"2024-04-25T01:22:57.727927+0000","last_became_peered":"2024-04-25T01:22:57.727927+0000","last_unstale":"2024-04-25T01:22:57.734032+0000","last_undegraded":"2024-04-25T01:22:57.734032+0000","last_fullsized":"2024-04-25T01:22:57.734032+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:51:28.858521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745825+0000","last_change":"2024-04-25T01:22:56.736642+0000","last_active":"2024-04-25T01:22:57.745825+0000","last_peered":"2024-04-25T01:22:57.745825+0000","last_clean":"2024-04-25T01:22:57.745825+0000","last_became_active":"2024-04-25T01:22:56.736373+0000","last_became_peered":"2024-04-25T01:22:56.736373+0000","last_unstale":"2024-04-25T01:22:57.745825+0000","last_undegraded":"2024-04-25T01:22:57.745825+0000","last_fullsized":"2024-04-25T01:22:57.745825+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:31:45.849059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.737656+0000","last_change":"2024-04-25T01:22:57.736069+0000","last_active":"2024-04-25T01:22:57.737656+0000","last_peered":"2024-04-25T01:22:57.737656+0000","last_clean":"2024-04-25T01:22:57.737656+0000","last_became_active":"2024-04-25T01:22:57.735818+0000","last_became_peered":"2024-04-25T01:22:57.735818+0000","last_unstale":"2024-04-25T01:22:57.737656+0000","last_undegraded":"2024-04-25T01:22:57.737656+0000","last_fullsized":"2024-04-25T01:22:57.737656+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:30:56.764099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732368+0000","last_change":"2024-04-25T01:22:56.736359+0000","last_active":"2024-04-25T01:22:57.732368+0000","last_peered":"2024-04-25T01:22:57.732368+0000","last_clean":"2024-04-25T01:22:57.732368+0000","last_became_active":"2024-04-25T01:22:56.736097+0000","last_became_peered":"2024-04-25T01:22:56.736097+0000","last_unstale":"2024-04-25T01:22:57.732368+0000","last_undegraded":"2024-04-25T01:22:57.732368+0000","last_fullsized":"2024-04-25T01:22:57.732368+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:03.652794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.744521+0000","last_change":"2024-04-25T01:22:57.737267+0000","last_active":"2024-04-25T01:22:57.744521+0000","last_peered":"2024-04-25T01:22:57.744521+0000","last_clean":"2024-04-25T01:22:57.744521+0000","last_became_active":"2024-04-25T01:22:57.737013+0000","last_became_peered":"2024-04-25T01:22:57.737013+0000","last_unstale":"2024-04-25T01:22:57.744521+0000","last_undegraded":"2024-04-25T01:22:57.744521+0000","last_fullsized":"2024-04-25T01:22:57.744521+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:04.656993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.727422+0000","last_change":"2024-04-25T01:22:56.735773+0000","last_active":"2024-04-25T01:22:57.727422+0000","last_peered":"2024-04-25T01:22:57.727422+0000","last_clean":"2024-04-25T01:22:57.727422+0000","last_became_active":"2024-04-25T01:22:56.735097+0000","last_became_peered":"2024-04-25T01:22:56.735097+0000","last_unstale":"2024-04-25T01:22:57.727422+0000","last_undegraded":"2024-04-25T01:22:57.727422+0000","last_fullsized":"2024-04-25T01:22:57.727422+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:55.266953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741142+0000","last_change":"2024-04-25T01:22:57.738267+0000","last_active":"2024-04-25T01:22:57.741142+0000","last_peered":"2024-04-25T01:22:57.741142+0000","last_clean":"2024-04-25T01:22:57.741142+0000","last_became_active":"2024-04-25T01:22:57.737985+0000","last_became_peered":"2024-04-25T01:22:57.737985+0000","last_unstale":"2024-04-25T01:22:57.741142+0000","last_undegraded":"2024-04-25T01:22:57.741142+0000","last_fullsized":"2024-04-25T01:22:57.741142+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:36.109995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":20,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747612+0000","last_change":"2024-04-25T01:22:56.733144+0000","last_active":"2024-04-25T01:22:57.747612+0000","last_peered":"2024-04-25T01:22:57.747612+0000","last_clean":"2024-04-25T01:22:57.747612+0000","last_became_active":"2024-04-25T01:22:56.732860+0000","last_became_peered":"2024-04-25T01:22:56.732860+0000","last_unstale":"2024-04-25T01:22:57.747612+0000","last_undegraded":"2024-04-25T01:22:57.747612+0000","last_fullsized":"2024-04-25T01:22:57.747612+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:51:27.854321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.742537+0000","last_change":"2024-04-25T01:22:57.741480+0000","last_active":"2024-04-25T01:22:57.742537+0000","last_peered":"2024-04-25T01:22:57.742537+0000","last_clean":"2024-04-25T01:22:57.742537+0000","last_became_active":"2024-04-25T01:22:57.741221+0000","last_became_peered":"2024-04-25T01:22:57.741221+0000","last_unstale":"2024-04-25T01:22:57.742537+0000","last_undegraded":"2024-04-25T01:22:57.742537+0000","last_fullsized":"2024-04-25T01:22:57.742537+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:17.967394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.742621+0000","last_change":"2024-04-25T01:22:56.734424+0000","last_active":"2024-04-25T01:22:57.742621+0000","last_peered":"2024-04-25T01:22:57.742621+0000","last_clean":"2024-04-25T01:22:57.742621+0000","last_became_active":"2024-04-25T01:22:56.734110+0000","last_became_peered":"2024-04-25T01:22:56.734110+0000","last_unstale":"2024-04-25T01:22:57.742621+0000","last_undegraded":"2024-04-25T01:22:57.742621+0000","last_fullsized":"2024-04-25T01:22:57.742621+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:32.437440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.741915+0000","last_change":"2024-04-25T01:22:57.738104+0000","last_active":"2024-04-25T01:22:57.741915+0000","last_peered":"2024-04-25T01:22:57.741915+0000","last_clean":"2024-04-25T01:22:57.741915+0000","last_became_active":"2024-04-25T01:22:57.737849+0000","last_became_peered":"2024-04-25T01:22:57.737849+0000","last_unstale":"2024-04-25T01:22:57.741915+0000","last_undegraded":"2024-04-25T01:22:57.741915+0000","last_fullsized":"2024-04-25T01:22:57.741915+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:51.207286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746874+0000","last_change":"2024-04-25T01:22:56.733102+0000","last_active":"2024-04-25T01:22:57.746874+0000","last_peered":"2024-04-25T01:22:57.746874+0000","last_clean":"2024-04-25T01:22:57.746874+0000","last_became_active":"2024-04-25T01:22:56.732817+0000","last_became_peered":"2024-04-25T01:22:56.732817+0000","last_unstale":"2024-04-25T01:22:57.746874+0000","last_undegraded":"2024-04-25T01:22:57.746874+0000","last_fullsized":"2024-04-25T01:22:57.746874+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:27:19.999586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.739883+0000","last_change":"2024-04-25T01:22:57.739111+0000","last_active":"2024-04-25T01:22:57.739883+0000","last_peered":"2024-04-25T01:22:57.739883+0000","last_clean":"2024-04-25T01:22:57.739883+0000","last_became_active":"2024-04-25T01:22:57.738736+0000","last_became_peered":"2024-04-25T01:22:57.738736+0000","last_unstale":"2024-04-25T01:22:57.739883+0000","last_undegraded":"2024-04-25T01:22:57.739883+0000","last_fullsized":"2024-04-25T01:22:57.739883+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:01.790499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746921+0000","last_change":"2024-04-25T01:22:56.732329+0000","last_active":"2024-04-25T01:22:57.746921+0000","last_peered":"2024-04-25T01:22:57.746921+0000","last_clean":"2024-04-25T01:22:57.746921+0000","last_became_active":"2024-04-25T01:22:56.732047+0000","last_became_peered":"2024-04-25T01:22:56.732047+0000","last_unstale":"2024-04-25T01:22:57.746921+0000","last_undegraded":"2024-04-25T01:22:57.746921+0000","last_fullsized":"2024-04-25T01:22:57.746921+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:10.714645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732925+0000","last_change":"2024-04-25T01:22:57.727455+0000","last_active":"2024-04-25T01:22:57.732925+0000","last_peered":"2024-04-25T01:22:57.732925+0000","last_clean":"2024-04-25T01:22:57.732925+0000","last_became_active":"2024-04-25T01:22:57.727177+0000","last_became_peered":"2024-04-25T01:22:57.727177+0000","last_unstale":"2024-04-25T01:22:57.732925+0000","last_undegraded":"2024-04-25T01:22:57.732925+0000","last_fullsized":"2024-04-25T01:22:57.732925+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:56.271152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731083+0000","last_change":"2024-04-25T01:22:56.735935+0000","last_active":"2024-04-25T01:22:57.731083+0000","last_peered":"2024-04-25T01:22:57.731083+0000","last_clean":"2024-04-25T01:22:57.731083+0000","last_became_active":"2024-04-25T01:22:56.735652+0000","last_became_peered":"2024-04-25T01:22:56.735652+0000","last_unstale":"2024-04-25T01:22:57.731083+0000","last_undegraded":"2024-04-25T01:22:57.731083+0000","last_fullsized":"2024-04-25T01:22:57.731083+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:51:27.854321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.743013+0000","last_change":"2024-04-25T01:22:57.739266+0000","last_active":"2024-04-25T01:22:57.743013+0000","last_peered":"2024-04-25T01:22:57.743013+0000","last_clean":"2024-04-25T01:22:57.743013+0000","last_became_active":"2024-04-25T01:22:57.739022+0000","last_became_peered":"2024-04-25T01:22:57.739022+0000","last_unstale":"2024-04-25T01:22:57.743013+0000","last_undegraded":"2024-04-25T01:22:57.743013+0000","last_fullsized":"2024-04-25T01:22:57.743013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:27.750839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.749579+0000","last_change":"2024-04-25T01:22:56.736970+0000","last_active":"2024-04-25T01:22:57.749579+0000","last_peered":"2024-04-25T01:22:57.749579+0000","last_clean":"2024-04-25T01:22:57.749579+0000","last_became_active":"2024-04-25T01:22:56.736705+0000","last_became_peered":"2024-04-25T01:22:56.736705+0000","last_unstale":"2024-04-25T01:22:57.749579+0000","last_undegraded":"2024-04-25T01:22:57.749579+0000","last_fullsized":"2024-04-25T01:22:57.749579+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:02:24.825202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.746427+0000","last_change":"2024-04-25T01:22:57.744271+0000","last_active":"2024-04-25T01:22:57.746427+0000","last_peered":"2024-04-25T01:22:57.746427+0000","last_clean":"2024-04-25T01:22:57.746427+0000","last_became_active":"2024-04-25T01:22:57.743996+0000","last_became_peered":"2024-04-25T01:22:57.743996+0000","last_unstale":"2024-04-25T01:22:57.746427+0000","last_undegraded":"2024-04-25T01:22:57.746427+0000","last_fullsized":"2024-04-25T01:22:57.746427+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:30.935419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.718229+0000","last_change":"2024-04-25T01:22:56.729921+0000","last_active":"2024-04-25T01:22:57.718229+0000","last_peered":"2024-04-25T01:22:57.718229+0000","last_clean":"2024-04-25T01:22:57.718229+0000","last_became_active":"2024-04-25T01:22:56.729647+0000","last_became_peered":"2024-04-25T01:22:56.729647+0000","last_unstale":"2024-04-25T01:22:57.718229+0000","last_undegraded":"2024-04-25T01:22:57.718229+0000","last_fullsized":"2024-04-25T01:22:57.718229+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:22.731853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.736403+0000","last_change":"2024-04-25T01:22:57.734499+0000","last_active":"2024-04-25T01:22:57.736403+0000","last_peered":"2024-04-25T01:22:57.736403+0000","last_clean":"2024-04-25T01:22:57.736403+0000","last_became_active":"2024-04-25T01:22:57.734241+0000","last_became_peered":"2024-04-25T01:22:57.734241+0000","last_unstale":"2024-04-25T01:22:57.736403+0000","last_undegraded":"2024-04-25T01:22:57.736403+0000","last_fullsized":"2024-04-25T01:22:57.736403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:24.178740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.725188+0000","last_change":"2024-04-25T01:22:56.734733+0000","last_active":"2024-04-25T01:22:57.725188+0000","last_peered":"2024-04-25T01:22:57.725188+0000","last_clean":"2024-04-25T01:22:57.725188+0000","last_became_active":"2024-04-25T01:22:56.734427+0000","last_became_peered":"2024-04-25T01:22:56.734427+0000","last_unstale":"2024-04-25T01:22:57.725188+0000","last_undegraded":"2024-04-25T01:22:57.725188+0000","last_fullsized":"2024-04-25T01:22:57.725188+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:13.942071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747199+0000","last_change":"2024-04-25T01:22:57.745963+0000","last_active":"2024-04-25T01:22:57.747199+0000","last_peered":"2024-04-25T01:22:57.747199+0000","last_clean":"2024-04-25T01:22:57.747199+0000","last_became_active":"2024-04-25T01:22:57.745221+0000","last_became_peered":"2024-04-25T01:22:57.745221+0000","last_unstale":"2024-04-25T01:22:57.747199+0000","last_undegraded":"2024-04-25T01:22:57.747199+0000","last_fullsized":"2024-04-25T01:22:57.747199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:34.172822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.718155+0000","last_change":"2024-04-25T01:22:56.732992+0000","last_active":"2024-04-25T01:22:57.718155+0000","last_peered":"2024-04-25T01:22:57.718155+0000","last_clean":"2024-04-25T01:22:57.718155+0000","last_became_active":"2024-04-25T01:22:56.732735+0000","last_became_peered":"2024-04-25T01:22:56.732735+0000","last_unstale":"2024-04-25T01:22:57.718155+0000","last_undegraded":"2024-04-25T01:22:57.718155+0000","last_fullsized":"2024-04-25T01:22:57.718155+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:25:03.869420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747371+0000","last_change":"2024-04-25T01:22:57.740010+0000","last_active":"2024-04-25T01:22:57.747371+0000","last_peered":"2024-04-25T01:22:57.747371+0000","last_clean":"2024-04-25T01:22:57.747371+0000","last_became_active":"2024-04-25T01:22:57.739731+0000","last_became_peered":"2024-04-25T01:22:57.739731+0000","last_unstale":"2024-04-25T01:22:57.747371+0000","last_undegraded":"2024-04-25T01:22:57.747371+0000","last_fullsized":"2024-04-25T01:22:57.747371+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:30.935419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747866+0000","last_change":"2024-04-25T01:22:56.736716+0000","last_active":"2024-04-25T01:22:57.747866+0000","last_peered":"2024-04-25T01:22:57.747866+0000","last_clean":"2024-04-25T01:22:57.747866+0000","last_became_active":"2024-04-25T01:22:56.736126+0000","last_became_peered":"2024-04-25T01:22:56.736126+0000","last_unstale":"2024-04-25T01:22:57.747866+0000","last_undegraded":"2024-04-25T01:22:57.747866+0000","last_fullsized":"2024-04-25T01:22:57.747866+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:38.801055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.740287+0000","last_change":"2024-04-25T01:22:57.738193+0000","last_active":"2024-04-25T01:22:57.740287+0000","last_peered":"2024-04-25T01:22:57.740287+0000","last_clean":"2024-04-25T01:22:57.740287+0000","last_became_active":"2024-04-25T01:22:57.737942+0000","last_became_peered":"2024-04-25T01:22:57.737942+0000","last_unstale":"2024-04-25T01:22:57.740287+0000","last_undegraded":"2024-04-25T01:22:57.740287+0000","last_fullsized":"2024-04-25T01:22:57.740287+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:33.441640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.718493+0000","last_change":"2024-04-25T01:22:56.728741+0000","last_active":"2024-04-25T01:22:57.718493+0000","last_peered":"2024-04-25T01:22:57.718493+0000","last_clean":"2024-04-25T01:22:57.718493+0000","last_became_active":"2024-04-25T01:22:56.728466+0000","last_became_peered":"2024-04-25T01:22:56.728466+0000","last_unstale":"2024-04-25T01:22:57.718493+0000","last_undegraded":"2024-04-25T01:22:57.718493+0000","last_fullsized":"2024-04-25T01:22:57.718493+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:36.492290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.731666+0000","last_change":"2024-04-25T01:22:57.725909+0000","last_active":"2024-04-25T01:22:57.731666+0000","last_peered":"2024-04-25T01:22:57.731666+0000","last_clean":"2024-04-25T01:22:57.731666+0000","last_became_active":"2024-04-25T01:22:57.725269+0000","last_became_peered":"2024-04-25T01:22:57.725269+0000","last_unstale":"2024-04-25T01:22:57.731666+0000","last_undegraded":"2024-04-25T01:22:57.731666+0000","last_fullsized":"2024-04-25T01:22:57.731666+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:24.178740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.726800+0000","last_change":"2024-04-25T01:22:56.734636+0000","last_active":"2024-04-25T01:22:57.726800+0000","last_peered":"2024-04-25T01:22:57.726800+0000","last_clean":"2024-04-25T01:22:57.726800+0000","last_became_active":"2024-04-25T01:22:56.734356+0000","last_became_peered":"2024-04-25T01:22:56.734356+0000","last_unstale":"2024-04-25T01:22:57.726800+0000","last_undegraded":"2024-04-25T01:22:57.726800+0000","last_fullsized":"2024-04-25T01:22:57.726800+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:19.967406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.734056+0000","last_change":"2024-04-25T01:22:57.728749+0000","last_active":"2024-04-25T01:22:57.734056+0000","last_peered":"2024-04-25T01:22:57.734056+0000","last_clean":"2024-04-25T01:22:57.734056+0000","last_became_active":"2024-04-25T01:22:57.728388+0000","last_became_peered":"2024-04-25T01:22:57.728388+0000","last_unstale":"2024-04-25T01:22:57.734056+0000","last_undegraded":"2024-04-25T01:22:57.734056+0000","last_fullsized":"2024-04-25T01:22:57.734056+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:01.790499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.728006+0000","last_change":"2024-04-25T01:22:56.735626+0000","last_active":"2024-04-25T01:22:57.728006+0000","last_peered":"2024-04-25T01:22:57.728006+0000","last_clean":"2024-04-25T01:22:57.728006+0000","last_became_active":"2024-04-25T01:22:56.735332+0000","last_became_peered":"2024-04-25T01:22:56.735332+0000","last_unstale":"2024-04-25T01:22:57.728006+0000","last_undegraded":"2024-04-25T01:22:57.728006+0000","last_fullsized":"2024-04-25T01:22:57.728006+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:13.942071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.747207+0000","last_change":"2024-04-25T01:22:57.738083+0000","last_active":"2024-04-25T01:22:57.747207+0000","last_peered":"2024-04-25T01:22:57.747207+0000","last_clean":"2024-04-25T01:22:57.747207+0000","last_became_active":"2024-04-25T01:22:57.737782+0000","last_became_peered":"2024-04-25T01:22:57.737782+0000","last_unstale":"2024-04-25T01:22:57.747207+0000","last_undegraded":"2024-04-25T01:22:57.747207+0000","last_fullsized":"2024-04-25T01:22:57.747207+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:33.441640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.728512+0000","last_change":"2024-04-25T01:22:56.738400+0000","last_active":"2024-04-25T01:22:57.728512+0000","last_peered":"2024-04-25T01:22:57.728512+0000","last_clean":"2024-04-25T01:22:57.728512+0000","last_became_active":"2024-04-25T01:22:56.738076+0000","last_became_peered":"2024-04-25T01:22:56.738076+0000","last_unstale":"2024-04-25T01:22:57.728512+0000","last_undegraded":"2024-04-25T01:22:57.728512+0000","last_fullsized":"2024-04-25T01:22:57.728512+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:51.358028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.732705+0000","last_change":"2024-04-25T01:22:57.727337+0000","last_active":"2024-04-25T01:22:57.732705+0000","last_peered":"2024-04-25T01:22:57.732705+0000","last_clean":"2024-04-25T01:22:57.732705+0000","last_became_active":"2024-04-25T01:22:57.727094+0000","last_became_peered":"2024-04-25T01:22:57.727094+0000","last_unstale":"2024-04-25T01:22:57.732705+0000","last_undegraded":"2024-04-25T01:22:57.732705+0000","last_fullsized":"2024-04-25T01:22:57.732705+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:32.419309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.721945+0000","last_change":"2024-04-25T01:22:56.738188+0000","last_active":"2024-04-25T01:22:57.721945+0000","last_peered":"2024-04-25T01:22:57.721945+0000","last_clean":"2024-04-25T01:22:57.721945+0000","last_became_active":"2024-04-25T01:22:56.737878+0000","last_became_peered":"2024-04-25T01:22:56.737878+0000","last_unstale":"2024-04-25T01:22:57.721945+0000","last_undegraded":"2024-04-25T01:22:57.721945+0000","last_fullsized":"2024-04-25T01:22:57.721945+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:51.358028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.745241+0000","last_change":"2024-04-25T01:22:57.739929+0000","last_active":"2024-04-25T01:22:57.745241+0000","last_peered":"2024-04-25T01:22:57.745241+0000","last_clean":"2024-04-25T01:22:57.745241+0000","last_became_active":"2024-04-25T01:22:57.739681+0000","last_became_peered":"2024-04-25T01:22:57.739681+0000","last_unstale":"2024-04-25T01:22:57.745241+0000","last_undegraded":"2024-04-25T01:22:57.745241+0000","last_fullsized":"2024-04-25T01:22:57.745241+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:20:31.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.721367+0000","last_change":"2024-04-25T01:22:56.728643+0000","last_active":"2024-04-25T01:22:57.721367+0000","last_peered":"2024-04-25T01:22:57.721367+0000","last_clean":"2024-04-25T01:22:57.721367+0000","last_became_active":"2024-04-25T01:22:56.728358+0000","last_became_peered":"2024-04-25T01:22:56.728358+0000","last_unstale":"2024-04-25T01:22:57.721367+0000","last_undegraded":"2024-04-25T01:22:57.721367+0000","last_fullsized":"2024-04-25T01:22:57.721367+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:45.986711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.738399+0000","last_change":"2024-04-25T01:22:57.733292+0000","last_active":"2024-04-25T01:22:57.738399+0000","last_peered":"2024-04-25T01:22:57.738399+0000","last_clean":"2024-04-25T01:22:57.738399+0000","last_became_active":"2024-04-25T01:22:57.733019+0000","last_became_peered":"2024-04-25T01:22:57.733019+0000","last_unstale":"2024-04-25T01:22:57.738399+0000","last_undegraded":"2024-04-25T01:22:57.738399+0000","last_fullsized":"2024-04-25T01:22:57.738399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.707505+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.707505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:12.932562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.729800+0000","last_change":"2024-04-25T01:22:56.734589+0000","last_active":"2024-04-25T01:22:57.729800+0000","last_peered":"2024-04-25T01:22:57.729800+0000","last_clean":"2024-04-25T01:22:57.729800+0000","last_became_active":"2024-04-25T01:22:56.734312+0000","last_became_peered":"2024-04-25T01:22:56.734312+0000","last_unstale":"2024-04-25T01:22:57.729800+0000","last_undegraded":"2024-04-25T01:22:57.729800+0000","last_fullsized":"2024-04-25T01:22:57.729800+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.703306+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.703306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:00.786299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":200,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T01:22:57.726586+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:22:57.726586+0000","last_peered":"2024-04-25T01:22:57.726586+0000","last_clean":"2024-04-25T01:22:57.726586+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:22:57.726586+0000","last_undegraded":"2024-04-25T01:22:57.726586+0000","last_fullsized":"2024-04-25T01:22:57.726586+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":183458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343832,"statfs":{"total":96636764160,"available":96608083968,"internally_reserved":0,"allocated":1007616,"data_stored":766569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:23:01.005 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T01:23:01.005 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T01:23:01.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.166+0000 7faa502ba640 1 Processor -- start 2024-04-25T01:23:01.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.166+0000 7faa502ba640 1 -- start start 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa502ba640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faa4806cb40 0x7faa48070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa502ba640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa48071110 0x7faa480715a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa502ba640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa480722c0 con 0x7faa4815f870 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa502ba640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7faa480725a0 con 0x7faa4806cb40 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa502ba640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faa480ca1d0 con 0x7faa48071110 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa4e056640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7faa4815f870 0x7faa4815fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58478/0 (socket says 172.21.15.77:58478) 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa4e056640 1 -- 172.21.15.77:0/518849371 learned_addr learned my addr 172.21.15.77:0/518849371 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa4d855640 1 --2- 172.21.15.77:0/518849371 >> v2:172.21.15.77:3300/0 conn(0x7faa4806cb40 0x7faa48070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa4e857640 1 --2- 172.21.15.77:0/518849371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa48071110 0x7faa480715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa4d054640 1 -- 172.21.15.77:0/518849371 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2480379000 0 0) 0x7faa480722c0 con 0x7faa4815f870 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa4d054640 1 -- 172.21.15.77:0/518849371 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa34007930 con 0x7faa4815f870 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa4d855640 1 -- 172.21.15.77:0/518849371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa48071110 msgr2=0x7faa480715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa4d855640 1 --2- 172.21.15.77:0/518849371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa48071110 0x7faa480715a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa4d855640 1 -- 172.21.15.77:0/518849371 >> v1:172.21.15.77:6789/0 conn(0x7faa4815f870 legacy=0x7faa4815fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.167+0000 7faa4d855640 1 -- 172.21.15.77:0/518849371 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa480ca4b0 con 0x7faa4806cb40 2024-04-25T01:23:01.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.168+0000 7faa4d855640 1 --2- 172.21.15.77:0/518849371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa4806cb40 0x7faa48070bd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7faa380097a0 tx=0x7faa3802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439cfc1c6064800b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:01.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.168+0000 7faa4d054640 1 -- 172.21.15.77:0/518849371 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa38037070 con 0x7faa4806cb40 2024-04-25T01:23:01.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.168+0000 7faa4d054640 1 -- 172.21.15.77:0/518849371 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa38032440 con 0x7faa4806cb40 2024-04-25T01:23:01.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.168+0000 7faa4d054640 1 -- 172.21.15.77:0/518849371 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa38032810 con 0x7faa4806cb40 2024-04-25T01:23:01.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.169+0000 7faa502ba640 1 -- 172.21.15.77:0/518849371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa4806cb40 msgr2=0x7faa48070bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.169+0000 7faa502ba640 1 --2- 172.21.15.77:0/518849371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa4806cb40 0x7faa48070bd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7faa380097a0 tx=0x7faa3802a180 comp rx=0 tx=0).stop 2024-04-25T01:23:01.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.169+0000 7faa502ba640 1 -- 172.21.15.77:0/518849371 shutdown_connections 2024-04-25T01:23:01.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.169+0000 7faa502ba640 1 --2- 172.21.15.77:0/518849371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa48071110 0x7faa480715a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.169+0000 7faa502ba640 1 --2- 172.21.15.77:0/518849371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa4806cb40 0x7faa48070bd0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.169+0000 7faa502ba640 1 -- 172.21.15.77:0/518849371 >> 172.21.15.77:0/518849371 conn(0x7faa480cfae0 msgr2=0x7faa480cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:01.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.169+0000 7faa502ba640 1 -- 172.21.15.77:0/518849371 shutdown_connections 2024-04-25T01:23:01.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.169+0000 7faa502ba640 1 -- 172.21.15.77:0/518849371 wait complete. 2024-04-25T01:23:01.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.170+0000 7faa502ba640 1 Processor -- start 2024-04-25T01:23:01.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.170+0000 7faa502ba640 1 -- start start 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa502ba640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa4806cb40 0x7faa480cf260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa502ba640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa48071110 0x7faa4814f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa502ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa4815f870 0x7faa480b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa502ba640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faa4806cfa0 con 0x7faa4815f870 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa502ba640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7faa48145910 con 0x7faa48071110 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa502ba640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faa480ce570 con 0x7faa4806cb40 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4e857640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa4815f870 0x7faa480b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4e857640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa4815f870 0x7faa480b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44138/0 (socket says 172.21.15.77:44138) 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4e857640 1 -- 172.21.15.77:0/1313215029 learned_addr learned my addr 172.21.15.77:0/1313215029 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4d855640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa48071110 0x7faa4814f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4e056640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa4806cb40 0x7faa480cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4e857640 1 -- 172.21.15.77:0/1313215029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa4806cb40 msgr2=0x7faa480cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4e857640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa4806cb40 0x7faa480cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4e857640 1 -- 172.21.15.77:0/1313215029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa48071110 msgr2=0x7faa4814f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4e857640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa48071110 0x7faa4814f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4e857640 1 -- 172.21.15.77:0/1313215029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4815b920 con 0x7faa4815f870 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4d855640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa48071110 0x7faa4814f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.171+0000 7faa4e857640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa4815f870 0x7faa480b4a00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7faa40007e40 tx=0x7faa4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.172+0000 7faa3effd640 1 -- 172.21.15.77:0/1313215029 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa40014620 con 0x7faa4815f870 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.172+0000 7faa3effd640 1 -- 172.21.15.77:0/1313215029 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa40019070 con 0x7faa4815f870 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.172+0000 7faa502ba640 1 -- 172.21.15.77:0/1313215029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa480cd960 con 0x7faa4815f870 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.172+0000 7faa3effd640 1 -- 172.21.15.77:0/1313215029 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa40014a20 con 0x7faa4815f870 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.172+0000 7faa502ba640 1 -- 172.21.15.77:0/1313215029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa480cdf20 con 0x7faa4815f870 2024-04-25T01:23:01.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.173+0000 7faa502ba640 1 -- 172.21.15.77:0/1313215029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa48068fb0 con 0x7faa4815f870 2024-04-25T01:23:01.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.173+0000 7faa3effd640 1 -- 172.21.15.77:0/1313215029 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7faa40014bc0 con 0x7faa4815f870 2024-04-25T01:23:01.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.174+0000 7faa3effd640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faa2c06be60 0x7faa2c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.174+0000 7faa3effd640 1 -- 172.21.15.77:0/1313215029 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7faa4008b8b0 con 0x7faa4815f870 2024-04-25T01:23:01.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.174+0000 7faa4e056640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faa2c06be60 0x7faa2c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.174+0000 7faa4e056640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faa2c06be60 0x7faa2c06e320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa44005880 tx=0x7faa44009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:01.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.178+0000 7faa3effd640 1 -- 172.21.15.77:0/1313215029 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7faa4005a0d0 con 0x7faa4815f870 2024-04-25T01:23:01.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.416+0000 7faa502ba640 1 -- 172.21.15.77:0/1313215029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7faa480cbeb0 con 0x7faa4815f870 2024-04-25T01:23:01.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.420+0000 7faa3effd640 1 -- 172.21.15.77:0/1313215029 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7faa400599d0 con 0x7faa4815f870 2024-04-25T01:23:01.401 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-04-25T01:23:01.401 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvrylmsfhcOhAAYZeVeMoyKmZdON0jUeFmeQ== 2024-04-25T01:23:01.401 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-04-25T01:23:01.401 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-04-25T01:23:01.401 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T01:23:01.401 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 -- 172.21.15.77:0/1313215029 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faa2c06be60 msgr2=0x7faa2c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faa2c06be60 0x7faa2c06e320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa44005880 tx=0x7faa44009040 comp rx=0 tx=0).stop 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 -- 172.21.15.77:0/1313215029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa4815f870 msgr2=0x7faa480b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa4815f870 0x7faa480b4a00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7faa40007e40 tx=0x7faa4000e4b0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 -- 172.21.15.77:0/1313215029 shutdown_connections 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faa2c06be60 0x7faa2c06e320 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa4815f870 0x7faa480b4a00 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa48071110 0x7faa4814f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 --2- 172.21.15.77:0/1313215029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa4806cb40 0x7faa480cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 -- 172.21.15.77:0/1313215029 >> 172.21.15.77:0/1313215029 conn(0x7faa480cfae0 msgr2=0x7faa480cab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 -- 172.21.15.77:0/1313215029 shutdown_connections 2024-04-25T01:23:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.422+0000 7faa502ba640 1 -- 172.21.15.77:0/1313215029 wait complete. 2024-04-25T01:23:01.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:01.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.530+0000 7ffaad5ed640 1 Processor -- start 2024-04-25T01:23:01.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.530+0000 7ffaad5ed640 1 -- start start 2024-04-25T01:23:01.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.530+0000 7ffaad5ed640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffaa80d2880 0x7ffaa80b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.530+0000 7ffaad5ed640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaa80d1120 con 0x7ffaa80b2da0 2024-04-25T01:23:01.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.530+0000 7ffaad5ed640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaa80d1400 con 0x7ffaa80d24a0 2024-04-25T01:23:01.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.530+0000 7ffaad5ed640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7ffaa80d16e0 con 0x7ffaa80d2880 2024-04-25T01:23:01.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.530+0000 7ffaa67fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffaa80d2880 0x7ffaa80b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.531+0000 7ffaa77fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ffaa80b2da0 0x7ffaa8161390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58480/0 (socket says 172.21.15.77:58480) 2024-04-25T01:23:01.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.531+0000 7ffaa77fe640 1 -- 172.21.15.77:0/4000891109 learned_addr learned my addr 172.21.15.77:0/4000891109 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:01.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.531+0000 7ffaa67fc640 1 -- 172.21.15.77:0/4000891109 >> v1:172.21.15.121:6789/0 conn(0x7ffaa80d24a0 legacy=0x7ffaa80d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:01.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.531+0000 7ffaa67fc640 1 -- 172.21.15.77:0/4000891109 >> v1:172.21.15.77:6789/0 conn(0x7ffaa80b2da0 legacy=0x7ffaa8161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.531+0000 7ffaa5ffb640 1 -- 172.21.15.77:0/4000891109 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2908032095 0 0) 0x7ffaa80d1120 con 0x7ffaa80b2da0 2024-04-25T01:23:01.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.531+0000 7ffaa67fc640 1 -- 172.21.15.77:0/4000891109 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa80cee60 con 0x7ffaa80d2880 2024-04-25T01:23:01.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.532+0000 7ffaa67fc640 1 --2- 172.21.15.77:0/4000891109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80d2880 0x7ffaa80b0bd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ffa900097a0 tx=0x7ffa9002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1abb8fd234d0a76c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:01.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.532+0000 7ffaa5ffb640 1 -- 172.21.15.77:0/4000891109 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa90037070 con 0x7ffaa80d2880 2024-04-25T01:23:01.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.532+0000 7ffaa5ffb640 1 -- 172.21.15.77:0/4000891109 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa90032440 con 0x7ffaa80d2880 2024-04-25T01:23:01.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.532+0000 7ffaa5ffb640 1 -- 172.21.15.77:0/4000891109 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa90032810 con 0x7ffaa80d2880 2024-04-25T01:23:01.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.532+0000 7ffaad5ed640 1 -- 172.21.15.77:0/4000891109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80d2880 msgr2=0x7ffaa80b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.532+0000 7ffaad5ed640 1 --2- 172.21.15.77:0/4000891109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80d2880 0x7ffaa80b0bd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ffa900097a0 tx=0x7ffa9002a1e0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.532+0000 7ffaad5ed640 1 -- 172.21.15.77:0/4000891109 shutdown_connections 2024-04-25T01:23:01.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.532+0000 7ffaad5ed640 1 --2- 172.21.15.77:0/4000891109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80d2880 0x7ffaa80b0bd0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.532+0000 7ffaad5ed640 1 -- 172.21.15.77:0/4000891109 >> 172.21.15.77:0/4000891109 conn(0x7ffaa809d200 msgr2=0x7ffaa80d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:01.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.532+0000 7ffaad5ed640 1 -- 172.21.15.77:0/4000891109 shutdown_connections 2024-04-25T01:23:01.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.532+0000 7ffaad5ed640 1 -- 172.21.15.77:0/4000891109 wait complete. 2024-04-25T01:23:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.533+0000 7ffaad5ed640 1 Processor -- start 2024-04-25T01:23:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.533+0000 7ffaad5ed640 1 -- start start 2024-04-25T01:23:01.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.534+0000 7ffaad5ed640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffaa80b2da0 0x7ffaa81f49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.534+0000 7ffaad5ed640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffaa80d24a0 0x7ffaa80b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.534+0000 7ffaad5ed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80d2880 0x7ffaa80b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.534+0000 7ffaad5ed640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa8161c20 con 0x7ffaa80d2880 2024-04-25T01:23:01.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.534+0000 7ffaad5ed640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa8161aa0 con 0x7ffaa80b2da0 2024-04-25T01:23:01.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.534+0000 7ffaad5ed640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa80cf380 con 0x7ffaa80d24a0 2024-04-25T01:23:01.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.534+0000 7ffaa77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80d2880 0x7ffaa80b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.534+0000 7ffaa77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80d2880 0x7ffaa80b51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44154/0 (socket says 172.21.15.77:44154) 2024-04-25T01:23:01.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.534+0000 7ffaa77fe640 1 -- 172.21.15.77:0/368358438 learned_addr learned my addr 172.21.15.77:0/368358438 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.534+0000 7ffaa67fc640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffaa80d24a0 0x7ffaa80b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.534+0000 7ffaa6ffd640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffaa80b2da0 0x7ffaa81f49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.535+0000 7ffaa77fe640 1 -- 172.21.15.77:0/368358438 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffaa80d24a0 msgr2=0x7ffaa80b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.535+0000 7ffaa77fe640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffaa80d24a0 0x7ffaa80b4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.535+0000 7ffaa77fe640 1 -- 172.21.15.77:0/368358438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffaa80b2da0 msgr2=0x7ffaa81f49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.535+0000 7ffaa77fe640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffaa80b2da0 0x7ffaa81f49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.535+0000 7ffaa77fe640 1 -- 172.21.15.77:0/368358438 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa80b3690 con 0x7ffaa80d2880 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.535+0000 7ffaa77fe640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80d2880 0x7ffaa80b51c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800c480 tx=0x7ffa9800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.535+0000 7ffa9f7fe640 1 -- 172.21.15.77:0/368358438 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98018070 con 0x7ffaa80d2880 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.535+0000 7ffa9f7fe640 1 -- 172.21.15.77:0/368358438 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa98007500 con 0x7ffaa80d2880 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.535+0000 7ffaad5ed640 1 -- 172.21.15.77:0/368358438 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa80b3970 con 0x7ffaa80d2880 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.535+0000 7ffa9f7fe640 1 -- 172.21.15.77:0/368358438 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98004a20 con 0x7ffaa80d2880 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.535+0000 7ffaad5ed640 1 -- 172.21.15.77:0/368358438 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa80b3f30 con 0x7ffaa80d2880 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.536+0000 7ffaad5ed640 1 -- 172.21.15.77:0/368358438 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa8068fb0 con 0x7ffaa80d2880 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.536+0000 7ffa9f7fe640 1 -- 172.21.15.77:0/368358438 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ffa9801b070 con 0x7ffaa80d2880 2024-04-25T01:23:01.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.537+0000 7ffa9f7fe640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa8806be30 0x7ffa8806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.537+0000 7ffa9f7fe640 1 -- 172.21.15.77:0/368358438 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7ffa9808a190 con 0x7ffaa80d2880 2024-04-25T01:23:01.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.537+0000 7ffaa6ffd640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa8806be30 0x7ffa8806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.537+0000 7ffaa6ffd640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa8806be30 0x7ffa8806e2f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80caa50 tx=0x7ffaa0003d90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.541+0000 7ffa9f7fe640 1 -- 172.21.15.77:0/368358438 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffa980583f0 con 0x7ffaa80d2880 2024-04-25T01:23:01.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.748+0000 7ffaad5ed640 1 -- 172.21.15.77:0/368358438 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffaa809d750 con 0x7ffaa80d2880 2024-04-25T01:23:01.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.748+0000 7ffa9f7fe640 1 -- 172.21.15.77:0/368358438 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5187 (secure 0 0 0) 0x7ffa9805c6f0 con 0x7ffaa80d2880 2024-04-25T01:23:01.729 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:01.729 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3139470828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3139470828},{"type":"v1","addr":"172.21.15.77:6835","nonce":3139470828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.066906+0000","modified":"2024-04-25T01:22:59.063205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5358":{"gid":5358,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2944320661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2944320661},{"type":"v1","addr":"172.21.15.77:6837","nonce":2944320661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5358,"qdb_cluster":[5358]},"id":2}]} 2024-04-25T01:23:01.729 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 16 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.750+0000 7ffaad5ed640 1 -- 172.21.15.77:0/368358438 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa8806be30 msgr2=0x7ffa8806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.750+0000 7ffaad5ed640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa8806be30 0x7ffa8806e2f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80caa50 tx=0x7ffaa0003d90 comp rx=0 tx=0).stop 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.750+0000 7ffaad5ed640 1 -- 172.21.15.77:0/368358438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80d2880 msgr2=0x7ffaa80b51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.750+0000 7ffaad5ed640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80d2880 0x7ffaa80b51c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800c480 tx=0x7ffa9800c940 comp rx=0 tx=0).stop 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.750+0000 7ffaad5ed640 1 -- 172.21.15.77:0/368358438 shutdown_connections 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.750+0000 7ffaad5ed640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ffa8806be30 0x7ffa8806e2f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.750+0000 7ffaad5ed640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80d2880 0x7ffaa80b51c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.750+0000 7ffaad5ed640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffaa80d24a0 0x7ffaa80b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.750+0000 7ffaad5ed640 1 --2- 172.21.15.77:0/368358438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffaa80b2da0 0x7ffaa81f49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.750+0000 7ffaad5ed640 1 -- 172.21.15.77:0/368358438 >> 172.21.15.77:0/368358438 conn(0x7ffaa809d200 msgr2=0x7ffaa815ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.750+0000 7ffaad5ed640 1 -- 172.21.15.77:0/368358438 shutdown_connections 2024-04-25T01:23:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:01.751+0000 7ffaad5ed640 1 -- 172.21.15.77:0/368358438 wait complete. 2024-04-25T01:23:02.144 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:22:57.066906+0000', 'modified': '2024-04-25T01:22:59.063205+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5358}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5358': {'gid': 5358, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2944320661', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2944320661}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2944320661}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5358, 'qdb_cluster': [5358]} 2024-04-25T01:23:02.144 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:23:02.144 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T01:23:02.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.263+0000 7f2a06534640 1 Processor -- start 2024-04-25T01:23:02.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.263+0000 7f2a06534640 1 -- start start 2024-04-25T01:23:02.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.263+0000 7f2a06534640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cfaf0 0x7f2a000d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.263+0000 7f2a06534640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a000d1e90 con 0x7f2a000d4a10 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.263+0000 7f2a06534640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a000b54a0 con 0x7f2a0015e4c0 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.263+0000 7f2a06534640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a000d2130 con 0x7f2a000cfaf0 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29ff7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2a000d4a10 0x7f2a000b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58496/0 (socket says 172.21.15.77:58496) 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29ff7fe640 1 -- 172.21.15.77:0/1938914055 learned_addr learned my addr 172.21.15.77:0/1938914055 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1554225169 0 0) 0x7f2a000d1e90 con 0x7f2a000d4a10 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29e8003520 con 0x7f2a000d4a10 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cfaf0 0x7f2a000d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3463661332 0 0) 0x7f29e8003520 con 0x7f2a000d4a10 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cfaf0 msgr2=0x7f2a000d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cfaf0 0x7f2a000d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 >> v1:172.21.15.121:6789/0 conn(0x7f2a0015e4c0 legacy=0x7f2a000d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a000b5780 con 0x7f2a000d4a10 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f29f4001670 con 0x7f2a000d4a10 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 >> v1:172.21.15.77:6789/0 conn(0x7f2a000d4a10 legacy=0x7f2a000b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 0x7f29e8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cfaf0 0x7f29e8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2a000d2130 con 0x7f29e80039e0 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f29e800baf0 con 0x7f29e8004800 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f29e800bcf0 con 0x7f2a000cfaf0 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29fffff640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 0x7f29e8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f29ff7fe640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.264+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cfaf0 0x7f29e8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.265+0000 7f29fffff640 1 -- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cfaf0 msgr2=0x7f29e8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.265+0000 7f29fffff640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cfaf0 0x7f29e8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.265+0000 7f29fffff640 1 -- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e8004800 msgr2=0x7f29e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.265+0000 7f29fffff640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.265+0000 7f29fffff640 1 -- 172.21.15.77:0/1938914055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e8003520 con 0x7f29e80039e0 2024-04-25T01:23:02.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.265+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cfaf0 0x7f29e8005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:02.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.265+0000 7f29fffff640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 0x7f29e8003e00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f29f0001750 tx=0x7f29e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64572ff99efbab1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.265+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f0001370 con 0x7f29e80039e0 2024-04-25T01:23:02.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.265+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29f00052d0 con 0x7f29e80039e0 2024-04-25T01:23:02.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.265+0000 7f29feffd640 1 -- 172.21.15.77:0/1938914055 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f002abe0 con 0x7f29e80039e0 2024-04-25T01:23:02.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.266+0000 7f2a06534640 1 -- 172.21.15.77:0/1938914055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 msgr2=0x7f29e8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.266+0000 7f2a06534640 1 --2- 172.21.15.77:0/1938914055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 0x7f29e8003e00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f29f0001750 tx=0x7f29e8005830 comp rx=0 tx=0).stop 2024-04-25T01:23:02.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.266+0000 7f2a04ad1640 1 -- 172.21.15.77:0/1938914055 reap_dead start 2024-04-25T01:23:02.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.266+0000 7f2a06534640 1 -- 172.21.15.77:0/1938914055 shutdown_connections 2024-04-25T01:23:02.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.266+0000 7f2a06534640 1 -- 172.21.15.77:0/1938914055 >> 172.21.15.77:0/1938914055 conn(0x7f2a000d3ef0 msgr2=0x7f2a000d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:02.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.266+0000 7f2a06534640 1 -- 172.21.15.77:0/1938914055 shutdown_connections 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.266+0000 7f2a06534640 1 -- 172.21.15.77:0/1938914055 wait complete. 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.267+0000 7f2a06534640 1 Processor -- start 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.267+0000 7f2a06534640 1 -- start start 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f2a06534640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 0x7f2a00153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f2a06534640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cd2b0 0x7f2a000cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f2a06534640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000c80c0 0x7f2a000c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f2a06534640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2a000b30a0 con 0x7f29e80039e0 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f2a06534640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2a000b2f20 con 0x7f2a000c80c0 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f2a06534640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a0015e8a0 con 0x7f2a000cd2b0 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f29fffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 0x7f2a00153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f29fffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 0x7f2a00153630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44172/0 (socket says 172.21.15.77:44172) 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f29fffff640 1 -- 172.21.15.77:0/2082924730 learned_addr learned my addr 172.21.15.77:0/2082924730 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f29fffff640 1 -- 172.21.15.77:0/2082924730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cd2b0 msgr2=0x7f2a000cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f29fffff640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cd2b0 0x7f2a000cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f29fffff640 1 -- 172.21.15.77:0/2082924730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000c80c0 msgr2=0x7f2a000c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f29fffff640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000c80c0 0x7f2a000c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f29fffff640 1 -- 172.21.15.77:0/2082924730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a000cbd20 con 0x7f29e80039e0 2024-04-25T01:23:02.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.268+0000 7f29fffff640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 0x7f2a00153630 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f29f0005040 tx=0x7f29f0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.269+0000 7f29fcff9640 1 -- 172.21.15.77:0/2082924730 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f0040020 con 0x7f29e80039e0 2024-04-25T01:23:02.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.269+0000 7f29fcff9640 1 -- 172.21.15.77:0/2082924730 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29f0005480 con 0x7f29e80039e0 2024-04-25T01:23:02.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.269+0000 7f29fcff9640 1 -- 172.21.15.77:0/2082924730 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f0007a30 con 0x7f29e80039e0 2024-04-25T01:23:02.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.269+0000 7f2a06534640 1 -- 172.21.15.77:0/2082924730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a000cc000 con 0x7f29e80039e0 2024-04-25T01:23:02.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.269+0000 7f2a06534640 1 -- 172.21.15.77:0/2082924730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a000cc5c0 con 0x7f29e80039e0 2024-04-25T01:23:02.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.270+0000 7f29fcff9640 1 -- 172.21.15.77:0/2082924730 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f29f0043020 con 0x7f29e80039e0 2024-04-25T01:23:02.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.271+0000 7f29fcff9640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29ec06be30 0x7f29ec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.271+0000 7f29fcff9640 1 -- 172.21.15.77:0/2082924730 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f29f00aae90 con 0x7f29e80039e0 2024-04-25T01:23:02.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.271+0000 7f29ff7fe640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29ec06be30 0x7f29ec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.271+0000 7f29ff7fe640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29ec06be30 0x7f29ec06e2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f29f40011e0 tx=0x7f29f4001130 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.271+0000 7f2a09241740 1 Processor -- start 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.271+0000 7f2a09241740 1 -- start start 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f2a09241740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae08408b60 0x55ae083bb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f2a09241740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083ebc80 0x55ae083eb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f2a09241740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083eb540 0x55ae083c0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f2a09241740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ae08323f00 con 0x55ae083eb540 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f2a09241740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ae082cc750 con 0x55ae08408b60 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f2a09241740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ae0831fec0 con 0x55ae083ebc80 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f2a04ad1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083eb540 0x55ae083c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f2a04ad1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083eb540 0x55ae083c0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44188/0 (socket says 172.21.15.77:44188) 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f2a04ad1640 1 -- 172.21.15.77:0/236960940 learned_addr learned my addr 172.21.15.77:0/236960940 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f29ff7fe640 1 --2- 172.21.15.77:0/236960940 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083ebc80 0x55ae083eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f29fffff640 1 --2- 172.21.15.77:0/236960940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae08408b60 0x55ae083bb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f2a04ad1640 1 -- 172.21.15.77:0/236960940 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083ebc80 msgr2=0x55ae083eb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.272+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/236960940 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083ebc80 0x55ae083eb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a04ad1640 1 -- 172.21.15.77:0/236960940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae08408b60 msgr2=0x55ae083bb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/236960940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae08408b60 0x55ae083bb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a04ad1640 1 -- 172.21.15.77:0/236960940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae080382d0 con 0x55ae083eb540 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f29ff7fe640 1 --2- 172.21.15.77:0/236960940 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083ebc80 0x55ae083eb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f29fffff640 1 --2- 172.21.15.77:0/236960940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae08408b60 0x55ae083bb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/236960940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083eb540 0x55ae083c0350 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009ea0 tx=0x7f29f8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ef66e276dd768 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f29e5ffb640 1 -- 172.21.15.77:0/236960940 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f8015640 con 0x55ae083eb540 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f29e5ffb640 1 -- 172.21.15.77:0/236960940 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29f801b070 con 0x55ae083eb540 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f29e5ffb640 1 -- 172.21.15.77:0/236960940 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f8015a20 con 0x55ae083eb540 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a09241740 1 -- 172.21.15.77:0/236960940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083eb540 msgr2=0x55ae083c0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a09241740 1 --2- 172.21.15.77:0/236960940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083eb540 0x55ae083c0350 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009ea0 tx=0x7f29f8009f90 comp rx=0 tx=0).stop 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a09241740 1 -- 172.21.15.77:0/236960940 shutdown_connections 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a09241740 1 --2- 172.21.15.77:0/236960940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083eb540 0x55ae083c0350 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a09241740 1 --2- 172.21.15.77:0/236960940 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083ebc80 0x55ae083eb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a09241740 1 --2- 172.21.15.77:0/236960940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae08408b60 0x55ae083bb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a09241740 1 -- 172.21.15.77:0/236960940 >> 172.21.15.77:0/236960940 conn(0x55ae083bd760 msgr2=0x55ae08406000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a09241740 1 -- 172.21.15.77:0/236960940 shutdown_connections 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.273+0000 7f2a09241740 1 -- 172.21.15.77:0/236960940 wait complete. 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.274+0000 7f2a09241740 1 -- start start 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.274+0000 7f2a09241740 1 Processor -- start 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f2a09241740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083ebc80 0x55ae083d89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f2a09241740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae08408b60 0x55ae083fdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f2a09241740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083fe160 0x55ae08407130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f2a09241740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ae083bf910 con 0x55ae083ebc80 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f2a09241740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ae082dfcf0 con 0x55ae083fe160 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f2a09241740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ae083bed00 con 0x55ae08408b60 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f2a09241740 10 client.0 _ll_register_callbacks cb 0x55ae082df6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f29fffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083ebc80 0x55ae083d89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f29fffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083ebc80 0x55ae083d89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44192/0 (socket says 172.21.15.77:44192) 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f29fffff640 1 -- 172.21.15.77:0/3421948089 learned_addr learned my addr 172.21.15.77:0/3421948089 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f29fffff640 1 -- 172.21.15.77:0/3421948089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae08408b60 msgr2=0x55ae083fdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f29fffff640 1 --2- 172.21.15.77:0/3421948089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae08408b60 0x55ae083fdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f29fffff640 1 -- 172.21.15.77:0/3421948089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083fe160 msgr2=0x55ae08407130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f29fffff640 1 --2- 172.21.15.77:0/3421948089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083fe160 0x55ae08407130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f29fffff640 1 -- 172.21.15.77:0/3421948089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae081026b0 con 0x55ae083ebc80 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f29fffff640 1 --2- 172.21.15.77:0/3421948089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083ebc80 0x55ae083d89a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f29f0078d40 tx=0x7f29f0079180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c762b727265ee82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f29deffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.275+0000 7f29deffd640 1 -- 172.21.15.77:0/3421948089 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f0078490 con 0x55ae083ebc80 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f29deffd640 1 -- 172.21.15.77:0/3421948089 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f29f0078630 con 0x55ae083ebc80 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f2a09241740 1 -- 172.21.15.77:0/3421948089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29f801b070 con 0x55ae083ebc80 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f29deffd640 1 -- 172.21.15.77:0/3421948089 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f29f0074d90 con 0x55ae083ebc80 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f2a09241740 10 client.5538 fetch_fsmap learned FSMap version 16 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f2a09241740 1 -- 172.21.15.77:0/3421948089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55ae083ecbb0 con 0x55ae083ebc80 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f29deffd640 1 -- 172.21.15.77:0/3421948089 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29f801b070 con 0x55ae083ebc80 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f29deffd640 1 -- 172.21.15.77:0/3421948089 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f29f0074f30 con 0x55ae083ebc80 2024-04-25T01:23:02.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f2a09241740 10 client.5538 fetch_fsmap finished waiting for FSMap version 16 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f2a09241740 10 client.5538 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f2a09241740 20 client.5538 populate_metadata read hostname 'smithi077' 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f2a09241740 1 --2- 172.21.15.77:0/3421948089 >> [v2:172.21.15.77:6836/2944320661,v1:172.21.15.77:6837/2944320661] conn(0x55ae084199f0 0x55ae08419dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f2a09241740 4 client.5538 mds_command: new command op to 5358 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f2a09241740 1 -- 172.21.15.77:0/3421948089 --> [v2:172.21.15.77:6836/2944320661,v1:172.21.15.77:6837/2944320661] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae08331440 con 0x55ae084199f0 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f29ff7fe640 1 --2- 172.21.15.77:0/3421948089 >> [v2:172.21.15.77:6836/2944320661,v1:172.21.15.77:6837/2944320661] conn(0x55ae084199f0 0x55ae08419dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f29deffd640 10 client.5538 ms_handle_connect on v2:172.21.15.77:6836/2944320661 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.276+0000 7f29ff7fe640 1 --2- 172.21.15.77:0/3421948089 >> [v2:172.21.15.77:6836/2944320661,v1:172.21.15.77:6837/2944320661] conn(0x55ae084199f0 0x55ae08419dd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3844becff1845524 server_cookie=304091ee7d57ef3c in_seq=0 out_seq=0 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f29deffd640 1 -- 172.21.15.77:0/3421948089 <== mds.0 v2:172.21.15.77:6836/2944320661 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ae08331440 con 0x55ae084199f0 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f29deffd640 10 client.5538 handle_command_reply: tid=0 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 1 client.5538 shutdown 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 1 -- 172.21.15.77:0/3421948089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083ebc80 msgr2=0x55ae083d89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 1 --2- 172.21.15.77:0/3421948089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083ebc80 0x55ae083d89a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f29f0078d40 tx=0x7f29f0079180 comp rx=0 tx=0).stop 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 1 -- 172.21.15.77:0/3421948089 shutdown_connections 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 1 --2- 172.21.15.77:0/3421948089 >> [v2:172.21.15.77:6836/2944320661,v1:172.21.15.77:6837/2944320661] conn(0x55ae084199f0 0x55ae08419dd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 1 --2- 172.21.15.77:0/3421948089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083fe160 0x55ae08407130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 1 --2- 172.21.15.77:0/3421948089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae08408b60 0x55ae083fdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f29deffd640 0 client.5538 ms_handle_reset on v2:172.21.15.77:6836/2944320661 2024-04-25T01:23:02.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 1 --2- 172.21.15.77:0/3421948089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae083ebc80 0x55ae083d89a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 1 -- 172.21.15.77:0/3421948089 >> 172.21.15.77:0/3421948089 conn(0x55ae083eb540 msgr2=0x55ae083babe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:02.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 1 -- 172.21.15.77:0/3421948089 shutdown_connections 2024-04-25T01:23:02.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 1 -- 172.21.15.77:0/3421948089 wait complete. 2024-04-25T01:23:02.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.277+0000 7f2a09241740 20 client.5538 trim_cache size 0 max 16384 2024-04-25T01:23:02.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.295+0000 7f2a09241740 1 Processor -- start 2024-04-25T01:23:02.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.295+0000 7f2a09241740 1 -- start start 2024-04-25T01:23:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a09241740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083ebc80 0x55ae083e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a09241740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083fe160 0x55ae083c0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a09241740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae08408030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a09241740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2a00200520 con 0x55ae08408b60 2024-04-25T01:23:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a09241740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2a000b9590 con 0x55ae083fe160 2024-04-25T01:23:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a09241740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a001ffed0 con 0x55ae083ebc80 2024-04-25T01:23:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a04ad1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae08408030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a04ad1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae08408030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44194/0 (socket says 172.21.15.77:44194) 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a04ad1640 1 -- 172.21.15.77:0/1648129309 learned_addr learned my addr 172.21.15.77:0/1648129309 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f29fffff640 1 --2- 172.21.15.77:0/1648129309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083ebc80 0x55ae083e6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f29ff7fe640 1 --2- 172.21.15.77:0/1648129309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083fe160 0x55ae083c0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a04ad1640 1 -- 172.21.15.77:0/1648129309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083ebc80 msgr2=0x55ae083e6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/1648129309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083ebc80 0x55ae083e6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a04ad1640 1 -- 172.21.15.77:0/1648129309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083fe160 msgr2=0x55ae083c0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/1648129309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083fe160 0x55ae083c0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f2a04ad1640 1 -- 172.21.15.77:0/1648129309 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae083e7110 con 0x55ae08408b60 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.296+0000 7f29ff7fe640 1 --2- 172.21.15.77:0/1648129309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083fe160 0x55ae083c0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/1648129309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae08408030 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009320 tx=0x7f29f8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50eac8ae8890a78a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f29e57fa640 1 -- 172.21.15.77:0/1648129309 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f8024070 con 0x55ae08408b60 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f29e57fa640 1 -- 172.21.15.77:0/1648129309 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29f801b040 con 0x55ae08408b60 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f29e57fa640 1 -- 172.21.15.77:0/1648129309 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f8015860 con 0x55ae08408b60 2024-04-25T01:23:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f2a09241740 1 -- 172.21.15.77:0/1648129309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 msgr2=0x55ae08408030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f2a09241740 1 --2- 172.21.15.77:0/1648129309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae08408030 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009320 tx=0x7f29f8013000 comp rx=0 tx=0).stop 2024-04-25T01:23:02.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f2a09241740 1 -- 172.21.15.77:0/1648129309 shutdown_connections 2024-04-25T01:23:02.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f2a09241740 1 --2- 172.21.15.77:0/1648129309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae08408030 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f2a09241740 1 --2- 172.21.15.77:0/1648129309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083fe160 0x55ae083c0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f2a09241740 1 --2- 172.21.15.77:0/1648129309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083ebc80 0x55ae083e6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f2a09241740 1 -- 172.21.15.77:0/1648129309 >> 172.21.15.77:0/1648129309 conn(0x55ae083eb540 msgr2=0x55ae083ed0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:02.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f2a09241740 1 -- 172.21.15.77:0/1648129309 shutdown_connections 2024-04-25T01:23:02.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.297+0000 7f2a09241740 1 -- 172.21.15.77:0/1648129309 wait complete. 2024-04-25T01:23:02.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.298+0000 7f2a09241740 1 -- start start 2024-04-25T01:23:02.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.298+0000 7f2a09241740 1 Processor -- start 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a09241740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083ebc80 0x55ae083bad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a09241740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083fe160 0x55ae08408400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a09241740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae084658c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a09241740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ae083eef60 con 0x55ae08408b60 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a09241740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ae083bf910 con 0x55ae083ebc80 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a09241740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ae083ee4e0 con 0x55ae083fe160 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a09241740 10 client.0 _ll_register_callbacks cb 0x7f29f0079830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a04ad1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae084658c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a04ad1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae084658c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44196/0 (socket says 172.21.15.77:44196) 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a04ad1640 1 -- 172.21.15.77:0/3840609013 learned_addr learned my addr 172.21.15.77:0/3840609013 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f29fffff640 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083ebc80 0x55ae083bad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f29ff7fe640 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083fe160 0x55ae08408400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a04ad1640 1 -- 172.21.15.77:0/3840609013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083fe160 msgr2=0x55ae08408400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083fe160 0x55ae08408400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a04ad1640 1 -- 172.21.15.77:0/3840609013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083ebc80 msgr2=0x55ae083bad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083ebc80 0x55ae083bad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a04ad1640 1 -- 172.21.15.77:0/3840609013 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae08408940 con 0x55ae08408b60 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f29fffff640 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083ebc80 0x55ae083bad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f29ff7fe640 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083fe160 0x55ae08408400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a04ad1640 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae084658c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f29f800b010 tx=0x7f29f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d25ccf4c4e5d02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f29deffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f29deffd640 1 -- 172.21.15.77:0/3840609013 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f8031030 con 0x55ae08408b60 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f29deffd640 1 -- 172.21.15.77:0/3840609013 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f29f8015650 con 0x55ae08408b60 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.299+0000 7f2a09241740 1 -- 172.21.15.77:0/3840609013 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29f801b040 con 0x55ae08408b60 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f29deffd640 1 -- 172.21.15.77:0/3840609013 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f29f803d050 con 0x55ae08408b60 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f2a09241740 10 client.5544 fetch_fsmap learned FSMap version 16 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f2a09241740 1 -- 172.21.15.77:0/3840609013 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55ae084181d0 con 0x55ae08408b60 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f29deffd640 1 -- 172.21.15.77:0/3840609013 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29f801b040 con 0x55ae08408b60 2024-04-25T01:23:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f29deffd640 1 -- 172.21.15.77:0/3840609013 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f29f8040020 con 0x55ae08408b60 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f2a09241740 10 client.5544 fetch_fsmap finished waiting for FSMap version 16 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f2a09241740 10 client.5544 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f2a09241740 20 client.5544 populate_metadata read hostname 'smithi077' 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f2a09241740 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.77:6836/2944320661,v1:172.21.15.77:6837/2944320661] conn(0x55ae08404860 0x55ae083bf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f2a09241740 4 client.5544 mds_command: new command op to 5358 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f2a09241740 1 -- 172.21.15.77:0/3840609013 --> [v2:172.21.15.77:6836/2944320661,v1:172.21.15.77:6837/2944320661] -- command(tid 0: {"prefix": "status"}) -- 0x55ae0832b790 con 0x55ae08404860 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f29fffff640 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.77:6836/2944320661,v1:172.21.15.77:6837/2944320661] conn(0x55ae08404860 0x55ae083bf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f29deffd640 10 client.5544 ms_handle_connect on v2:172.21.15.77:6836/2944320661 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f29fffff640 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.77:6836/2944320661,v1:172.21.15.77:6837/2944320661] conn(0x55ae08404860 0x55ae083bf910 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b40babc40447b887 server_cookie=f72c8f0340136df in_seq=0 out_seq=0 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f29deffd640 1 -- 172.21.15.77:0/3840609013 <== mds.0 v2:172.21.15.77:6836/2944320661 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55ae0832b790 con 0x55ae08404860 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f29deffd640 10 client.5544 handle_command_reply: tid=0 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.300+0000 7f2a09241740 1 client.5544 shutdown 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a09241740 1 -- 172.21.15.77:0/3840609013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 msgr2=0x55ae084658c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a09241740 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae084658c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f29f800b010 tx=0x7f29f800b040 comp rx=0 tx=0).stop 2024-04-25T01:23:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a09241740 1 -- 172.21.15.77:0/3840609013 shutdown_connections 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a09241740 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.77:6836/2944320661,v1:172.21.15.77:6837/2944320661] conn(0x55ae08404860 0x55ae083bf910 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a09241740 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae08408b60 0x55ae084658c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f29deffd640 0 client.5544 ms_handle_reset on v2:172.21.15.77:6836/2944320661 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a09241740 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae083fe160 0x55ae08408400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a09241740 1 --2- 172.21.15.77:0/3840609013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae083ebc80 0x55ae083bad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a09241740 1 -- 172.21.15.77:0/3840609013 >> 172.21.15.77:0/3840609013 conn(0x55ae083eb540 msgr2=0x55ae083bc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a09241740 1 -- 172.21.15.77:0/3840609013 shutdown_connections 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a09241740 1 -- 172.21.15.77:0/3840609013 wait complete. 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a09241740 20 client.5544 trim_cache size 0 max 16384 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "7c3c39ee-077e-476c-8994-eede4fab301d", 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-04-25T01:23:02.287 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 5358, 2024-04-25T01:23:02.288 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-04-25T01:23:02.288 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-04-25T01:23:02.288 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-04-25T01:23:02.288 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 4.5750967329999996, 2024-04-25T01:23:02.288 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 16, 2024-04-25T01:23:02.288 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 38, 2024-04-25T01:23:02.288 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 38, 2024-04-25T01:23:02.288 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 11.468486798000001 2024-04-25T01:23:02.288 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a06534640 1 -- 172.21.15.77:0/2082924730 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29ec06be30 msgr2=0x7f29ec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a06534640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29ec06be30 0x7f29ec06e2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f29f40011e0 tx=0x7f29f4001130 comp rx=0 tx=0).stop 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a06534640 1 -- 172.21.15.77:0/2082924730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 msgr2=0x7f2a00153630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a06534640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 0x7f2a00153630 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f29f0005040 tx=0x7f29f0001120 comp rx=0 tx=0).stop 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a06534640 1 -- 172.21.15.77:0/2082924730 shutdown_connections 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a06534640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29ec06be30 0x7f29ec06e2f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a06534640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000c80c0 0x7f2a000c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a06534640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a000cd2b0 0x7f2a000cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a06534640 1 --2- 172.21.15.77:0/2082924730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e80039e0 0x7f2a00153630 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.301+0000 7f2a06534640 1 -- 172.21.15.77:0/2082924730 >> 172.21.15.77:0/2082924730 conn(0x7f2a000d3ef0 msgr2=0x7f2a000b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.302+0000 7f2a06534640 1 -- 172.21.15.77:0/2082924730 shutdown_connections 2024-04-25T01:23:02.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.302+0000 7f2a06534640 1 -- 172.21.15.77:0/2082924730 wait complete. 2024-04-25T01:23:02.292 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:23:02.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.408+0000 7f35f7e76640 1 Processor -- start 2024-04-25T01:23:02.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.408+0000 7f35f7e76640 1 -- start start 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f7e76640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f35f0154ac0 0x7f35f015b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f7e76640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f35f006e550 0x7f35f006e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f7e76640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35f0154ea0 con 0x7f35f00cf2a0 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f7e76640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f35f006d540 con 0x7f35f006e550 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f7e76640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f35f006eea0 con 0x7f35f0154ac0 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f6413640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f35f006e550 0x7f35f006e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f5411640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f35f00cf2a0 0x7f35f00711f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58512/0 (socket says 172.21.15.77:58512) 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f5411640 1 -- 172.21.15.77:0/3192463739 learned_addr learned my addr 172.21.15.77:0/3192463739 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f5c12640 1 --2- 172.21.15.77:0/3192463739 >> v2:172.21.15.121:3300/0 conn(0x7f35f0154ac0 0x7f35f015b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f6413640 1 -- 172.21.15.77:0/3192463739 >> v2:172.21.15.121:3300/0 conn(0x7f35f0154ac0 msgr2=0x7f35f015b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f6413640 1 --2- 172.21.15.77:0/3192463739 >> v2:172.21.15.121:3300/0 conn(0x7f35f0154ac0 0x7f35f015b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f6413640 1 -- 172.21.15.77:0/3192463739 >> v1:172.21.15.77:6789/0 conn(0x7f35f00cf2a0 legacy=0x7f35f00711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f6413640 1 -- 172.21.15.77:0/3192463739 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f006d790 con 0x7f35f006e550 2024-04-25T01:23:02.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.409+0000 7f35f4c10640 1 -- 172.21.15.77:0/3192463739 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3122816508 0 0) 0x7f35f0154ea0 con 0x7f35f00cf2a0 2024-04-25T01:23:02.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.410+0000 7f35f6413640 1 --2- 172.21.15.77:0/3192463739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35f006e550 0x7f35f006e960 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f35f00d0bc0 tx=0x7f35e802b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50896295955bd72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.410+0000 7f35f4c10640 1 -- 172.21.15.77:0/3192463739 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8008640 con 0x7f35f006e550 2024-04-25T01:23:02.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.410+0000 7f35f4c10640 1 -- 172.21.15.77:0/3192463739 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35e80087e0 con 0x7f35f006e550 2024-04-25T01:23:02.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.410+0000 7f35f4c10640 1 -- 172.21.15.77:0/3192463739 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8008bb0 con 0x7f35f006e550 2024-04-25T01:23:02.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.410+0000 7f35f7e76640 1 -- 172.21.15.77:0/3192463739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35f006e550 msgr2=0x7f35f006e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.410+0000 7f35f7e76640 1 --2- 172.21.15.77:0/3192463739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35f006e550 0x7f35f006e960 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f35f00d0bc0 tx=0x7f35e802b780 comp rx=0 tx=0).stop 2024-04-25T01:23:02.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.411+0000 7f35f7e76640 1 -- 172.21.15.77:0/3192463739 shutdown_connections 2024-04-25T01:23:02.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.411+0000 7f35f7e76640 1 --2- 172.21.15.77:0/3192463739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35f006e550 0x7f35f006e960 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.411+0000 7f35f7e76640 1 --2- 172.21.15.77:0/3192463739 >> v2:172.21.15.121:3300/0 conn(0x7f35f0154ac0 0x7f35f015b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.411+0000 7f35f7e76640 1 -- 172.21.15.77:0/3192463739 >> 172.21.15.77:0/3192463739 conn(0x7f35f00b1820 msgr2=0x7f35f00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:02.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.411+0000 7f35f7e76640 1 -- 172.21.15.77:0/3192463739 shutdown_connections 2024-04-25T01:23:02.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.411+0000 7f35f7e76640 1 -- 172.21.15.77:0/3192463739 wait complete. 2024-04-25T01:23:02.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.412+0000 7f35f7e76640 1 Processor -- start 2024-04-25T01:23:02.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.412+0000 7f35f7e76640 1 -- start start 2024-04-25T01:23:02.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.412+0000 7f35f7e76640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35f006e550 0x7f35f00ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.412+0000 7f35f7e76640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35f00cf2a0 0x7f35f0150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.412+0000 7f35f7e76640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35f0154ac0 0x7f35f01f1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.412+0000 7f35f7e76640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f35f006e030 con 0x7f35f006e550 2024-04-25T01:23:02.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.412+0000 7f35f7e76640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f35f006dd50 con 0x7f35f0154ac0 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.412+0000 7f35f7e76640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f35f0071900 con 0x7f35f00cf2a0 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f5c12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35f006e550 0x7f35f00ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f5c12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35f006e550 0x7f35f00ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44214/0 (socket says 172.21.15.77:44214) 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f5c12640 1 -- 172.21.15.77:0/173160586 learned_addr learned my addr 172.21.15.77:0/173160586 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f5411640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35f00cf2a0 0x7f35f0150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f6413640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35f0154ac0 0x7f35f01f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f5c12640 1 -- 172.21.15.77:0/173160586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35f00cf2a0 msgr2=0x7f35f0150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f5c12640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35f00cf2a0 0x7f35f0150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f5c12640 1 -- 172.21.15.77:0/173160586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35f0154ac0 msgr2=0x7f35f01f1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f5c12640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35f0154ac0 0x7f35f01f1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f5c12640 1 -- 172.21.15.77:0/173160586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f0151280 con 0x7f35f006e550 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f5411640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35f00cf2a0 0x7f35f0150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f6413640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35f0154ac0 0x7f35f01f1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f5c12640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35f006e550 0x7f35f00ce260 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00d620 tx=0x7f35ec00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35e67fc640 1 -- 172.21.15.77:0/173160586 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec00b620 con 0x7f35f006e550 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35e67fc640 1 -- 172.21.15.77:0/173160586 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35ec019070 con 0x7f35f006e550 2024-04-25T01:23:02.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f7e76640 1 -- 172.21.15.77:0/173160586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f01f2180 con 0x7f35f006e550 2024-04-25T01:23:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35e67fc640 1 -- 172.21.15.77:0/173160586 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec00ba00 con 0x7f35f006e550 2024-04-25T01:23:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.413+0000 7f35f7e76640 1 -- 172.21.15.77:0/173160586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f01f2740 con 0x7f35f006e550 2024-04-25T01:23:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.414+0000 7f35f7e76640 1 -- 172.21.15.77:0/173160586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35f0068fb0 con 0x7f35f006e550 2024-04-25T01:23:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.415+0000 7f35e67fc640 1 -- 172.21.15.77:0/173160586 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f35ec00bba0 con 0x7f35f006e550 2024-04-25T01:23:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.415+0000 7f35e67fc640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f35d406be30 0x7f35d406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.415+0000 7f35e67fc640 1 -- 172.21.15.77:0/173160586 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f35ec08b420 con 0x7f35f006e550 2024-04-25T01:23:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.415+0000 7f35f5411640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f35d406be30 0x7f35d406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.415+0000 7f35f5411640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f35d406be30 0x7f35d406e2f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f35f0154490 tx=0x7f35e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.419+0000 7f35e67fc640 1 -- 172.21.15.77:0/173160586 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f35ec059c40 con 0x7f35f006e550 2024-04-25T01:23:02.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.628+0000 7f35f7e76640 1 -- 172.21.15.77:0/173160586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f35f015b3d0 con 0x7f35f006e550 2024-04-25T01:23:02.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.629+0000 7f35e67fc640 1 -- 172.21.15.77:0/173160586 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f35ec059540 con 0x7f35f006e550 2024-04-25T01:23:02.609 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T01:23:02.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 -- 172.21.15.77:0/173160586 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f35d406be30 msgr2=0x7f35d406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f35d406be30 0x7f35d406e2f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f35f0154490 tx=0x7f35e0009040 comp rx=0 tx=0).stop 2024-04-25T01:23:02.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 -- 172.21.15.77:0/173160586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35f006e550 msgr2=0x7f35f00ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35f006e550 0x7f35f00ce260 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00d620 tx=0x7f35ec00dae0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 -- 172.21.15.77:0/173160586 shutdown_connections 2024-04-25T01:23:02.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f35d406be30 0x7f35d406e2f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35f0154ac0 0x7f35f01f1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35f00cf2a0 0x7f35f0150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 --2- 172.21.15.77:0/173160586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35f006e550 0x7f35f00ce260 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 -- 172.21.15.77:0/173160586 >> 172.21.15.77:0/173160586 conn(0x7f35f00b1820 msgr2=0x7f35f00711f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:02.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 -- 172.21.15.77:0/173160586 shutdown_connections 2024-04-25T01:23:02.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02.631+0000 7f35f7e76640 1 -- 172.21.15.77:0/173160586 wait complete. 2024-04-25T01:23:02.621 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T01:23:02.622 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:23:02.622 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-04-25T01:23:02.622 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:23:02.622 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:02.622 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T01:23:02.622 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T01:23:02.622 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:23:02.622 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:23:02.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:02.622 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-04-25T01:23:02.637 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:23:02.637 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:23:02.637 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:23:02.637 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:02.637 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-04-25T01:23:02.637 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:02.637 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:02.637 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42666sec preferred_lft 42666sec 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a9ad:d2f:d456:e65f/64 scope link noprefixroute 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0a:90:ec:3f:d1:57 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:02.638 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::b40c:78ff:fecf:cd0c/64 scope link 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run.smithi077.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0a:90:ec:3f:d1:57 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::890:ecff:fe3f:d157/64 scope link 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run.smithi077.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 12:78:6c:75:3b:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1078:6cff:fe75:3be8/64 scope link 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:02.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:02.639 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:23:02.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2024-04-25T01:23:02.704+0000 7faad970f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:23:02.699 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-04-25T01:23:02.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:02.700 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-04-25T01:23:02.756 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-04-25T01:23:02.756 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:23:02.756 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:23:02.756 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:23:02.756 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:23:02.756 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:23:02.756 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:23:02.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:23:02.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:23:02.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:23:02.757 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:23:02.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:23:02.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:02.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:02.879 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-04-25T01:23:02.894 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T01:23:02.894 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:23:02.894 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:23:02.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:02.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-04-25T01:23:02.957 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:23:02.958 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:03.012 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:23:03.012 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:23:03.012 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:03.066 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:23:03.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:03.066 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:03.147 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:03.147 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:23:03.147 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:23:03.155 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:03.156 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:23:03.156 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:23:03.156 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:23:03.156 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:23:03.160 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:03.160 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:23:03.161 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:23:03.161 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:23:03.161 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:23:03.161 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790" 2024-04-25T01:23:03.161 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790:/ 2024-04-25T01:23:03.161 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=7c3c39ee-077e-476c-8994-eede4fab301d" will pass to kernel 2024-04-25T01:23:03.164 INFO:teuthology.orchestra.run.smithi077.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T01:23:03.164 INFO:teuthology.orchestra.run.smithi077.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:23:03.164 INFO:teuthology.orchestra.run.smithi077.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:23:03.164 INFO:teuthology.orchestra.run.smithi077.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:23:03.164 INFO:teuthology.orchestra.run.smithi077.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:23:03.165 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:23:03.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:03.165 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:03.235 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:23:03.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:23:03.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:03.335 INFO:teuthology.orchestra.run.smithi077.stdout:client5556 2024-04-25T01:23:03.335 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:23:03.335 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:03.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:03.351 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir1) 2024-04-25T01:23:03.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:03.412 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch testdir1/testfile) 2024-04-25T01:23:03.468 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-04-25T01:23:03.468 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-04-25T01:23:03.468 DEBUG:teuthology.orchestra.run.smithi077:> import stat 2024-04-25T01:23:03.469 DEBUG:teuthology.orchestra.run.smithi077:> 2024-04-25T01:23:03.469 DEBUG:teuthology.orchestra.run.smithi077:> print(os.stat("/home/ubuntu/cephtest/mnt.0/testdir1").st_ino) 2024-04-25T01:23:03.469 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-04-25T01:23:03.572 INFO:teuthology.orchestra.run.smithi077.stdout:1099511627776 2024-04-25T01:23:03.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:03.581 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:23:03.581 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:03.636 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:23:03.636 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:03.692 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:03.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.848+0000 7f2833f42640 1 Processor -- start 2024-04-25T01:23:03.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.849+0000 7f2833f42640 1 -- start start 2024-04-25T01:23:03.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.849+0000 7f2833f42640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f282c0d4a10 0x7f282c0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.849+0000 7f2833f42640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f282c0d1e90 con 0x7f282c15e4c0 2024-04-25T01:23:03.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.849+0000 7f2833f42640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f282c0b54a0 con 0x7f282c0b3ee0 2024-04-25T01:23:03.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.849+0000 7f2833f42640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f282c0d2130 con 0x7f282c0d4a10 2024-04-25T01:23:03.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.849+0000 7f28314dd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f282c0d4a10 0x7f282c0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.849+0000 7f28314dd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f282c0d4a10 0x7f282c0cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44216/0 (socket says 172.21.15.77:44216) 2024-04-25T01:23:03.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.849+0000 7f2831cde640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f282c15e4c0 0x7f282c0d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58528/0 (socket says 172.21.15.77:58528) 2024-04-25T01:23:03.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.850+0000 7f28314dd640 1 -- 172.21.15.77:0/1567147224 learned_addr learned my addr 172.21.15.77:0/1567147224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:03.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.850+0000 7f28314dd640 1 -- 172.21.15.77:0/1567147224 >> v1:172.21.15.121:6789/0 conn(0x7f282c0b3ee0 legacy=0x7f282c0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:03.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.850+0000 7f28314dd640 1 -- 172.21.15.77:0/1567147224 >> v1:172.21.15.77:6789/0 conn(0x7f282c15e4c0 legacy=0x7f282c0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.850+0000 7f28314dd640 1 -- 172.21.15.77:0/1567147224 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282c0b5780 con 0x7f282c0d4a10 2024-04-25T01:23:03.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.850+0000 7f2830cdc640 1 -- 172.21.15.77:0/1567147224 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847036594 0 0) 0x7f282c0d1e90 con 0x7f282c15e4c0 2024-04-25T01:23:03.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.850+0000 7f28314dd640 1 --2- 172.21.15.77:0/1567147224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f282c0d4a10 0x7f282c0cfaf0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f281c009870 tx=0x7f281c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4910899a972ea6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.851+0000 7f2830cdc640 1 -- 172.21.15.77:0/1567147224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c037070 con 0x7f282c0d4a10 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.851+0000 7f2830cdc640 1 -- 172.21.15.77:0/1567147224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f281c032440 con 0x7f282c0d4a10 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.851+0000 7f2830cdc640 1 -- 172.21.15.77:0/1567147224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c032810 con 0x7f282c0d4a10 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.851+0000 7f2833f42640 1 -- 172.21.15.77:0/1567147224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f282c0d4a10 msgr2=0x7f282c0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.851+0000 7f2833f42640 1 --2- 172.21.15.77:0/1567147224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f282c0d4a10 0x7f282c0cfaf0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f281c009870 tx=0x7f281c02a210 comp rx=0 tx=0).stop 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.851+0000 7f2833f42640 1 -- 172.21.15.77:0/1567147224 shutdown_connections 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.851+0000 7f2833f42640 1 --2- 172.21.15.77:0/1567147224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f282c0d4a10 0x7f282c0cfaf0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.851+0000 7f2833f42640 1 -- 172.21.15.77:0/1567147224 >> 172.21.15.77:0/1567147224 conn(0x7f282c0d3ef0 msgr2=0x7f282c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.851+0000 7f2833f42640 1 -- 172.21.15.77:0/1567147224 shutdown_connections 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.851+0000 7f2833f42640 1 -- 172.21.15.77:0/1567147224 wait complete. 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.852+0000 7f2833f42640 1 Processor -- start 2024-04-25T01:23:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.852+0000 7f2833f42640 1 -- start start 2024-04-25T01:23:03.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2833f42640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f282c0b3ee0 0x7f282c159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2833f42640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f282c0d4a10 0x7f282c0b8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2833f42640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f282c15e4c0 0x7f282c0b96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2833f42640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f282c0b30a0 con 0x7f282c0b3ee0 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2833f42640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f282c0b2f20 con 0x7f282c15e4c0 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2833f42640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f282c15e8a0 con 0x7f282c0d4a10 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2831cde640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f282c0b3ee0 0x7f282c159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2831cde640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f282c0b3ee0 0x7f282c159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44226/0 (socket says 172.21.15.77:44226) 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2831cde640 1 -- 172.21.15.77:0/3449634549 learned_addr learned my addr 172.21.15.77:0/3449634549 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f28324df640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f282c15e4c0 0x7f282c0b96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f28314dd640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f282c0d4a10 0x7f282c0b8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2831cde640 1 -- 172.21.15.77:0/3449634549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f282c0d4a10 msgr2=0x7f282c0b8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2831cde640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f282c0d4a10 0x7f282c0b8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2831cde640 1 -- 172.21.15.77:0/3449634549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f282c15e4c0 msgr2=0x7f282c0b96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2831cde640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f282c15e4c0 0x7f282c0b96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f2831cde640 1 -- 172.21.15.77:0/3449634549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282c0b7bc0 con 0x7f282c0b3ee0 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.853+0000 7f28324df640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f282c15e4c0 0x7f282c0b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.854+0000 7f2831cde640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f282c0b3ee0 0x7f282c159fa0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f282800c500 tx=0x7f282800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.854+0000 7f28227fc640 1 -- 172.21.15.77:0/3449634549 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2828018070 con 0x7f282c0b3ee0 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.854+0000 7f28227fc640 1 -- 172.21.15.77:0/3449634549 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2828007580 con 0x7f282c0b3ee0 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.854+0000 7f28227fc640 1 -- 172.21.15.77:0/3449634549 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2828004a20 con 0x7f282c0b3ee0 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.854+0000 7f2833f42640 1 -- 172.21.15.77:0/3449634549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f282c0b7ea0 con 0x7f282c0b3ee0 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.854+0000 7f2833f42640 1 -- 172.21.15.77:0/3449634549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f282c1f7b30 con 0x7f282c0b3ee0 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.855+0000 7f2833f42640 1 -- 172.21.15.77:0/3449634549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f282c068fb0 con 0x7f282c0b3ee0 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.855+0000 7f28227fc640 1 -- 172.21.15.77:0/3449634549 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f282801b070 con 0x7f282c0b3ee0 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.856+0000 7f28227fc640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f281006be30 0x7f281006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.856+0000 7f28227fc640 1 -- 172.21.15.77:0/3449634549 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f282808a2d0 con 0x7f282c0b3ee0 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.856+0000 7f28314dd640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f281006be30 0x7f281006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.856+0000 7f28314dd640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f281006be30 0x7f281006e2f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f281c034000 tx=0x7f281c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:03.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:03.860+0000 7f28227fc640 1 -- 172.21.15.77:0/3449634549 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2828058b80 con 0x7f282c0b3ee0 2024-04-25T01:23:04.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.032+0000 7f2833f42640 1 -- 172.21.15.77:0/3449634549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f282c0b8560 con 0x7f282c0b3ee0 2024-04-25T01:23:04.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.033+0000 7f28227fc640 1 -- 172.21.15.77:0/3449634549 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16942 (secure 0 0 0) 0x7f2828058480 con 0x7f282c0b3ee0 2024-04-25T01:23:04.014 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:04.014 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":38,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:22:58.725224+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.457887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.029883+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:04.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 -- 172.21.15.77:0/3449634549 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f281006be30 msgr2=0x7f281006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f281006be30 0x7f281006e2f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f281c034000 tx=0x7f281c02e000 comp rx=0 tx=0).stop 2024-04-25T01:23:04.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 -- 172.21.15.77:0/3449634549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f282c0b3ee0 msgr2=0x7f282c159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f282c0b3ee0 0x7f282c159fa0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f282800c500 tx=0x7f282800c9c0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 -- 172.21.15.77:0/3449634549 shutdown_connections 2024-04-25T01:23:04.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f281006be30 0x7f281006e2f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f282c15e4c0 0x7f282c0b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f282c0d4a10 0x7f282c0b8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 --2- 172.21.15.77:0/3449634549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f282c0b3ee0 0x7f282c159fa0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 -- 172.21.15.77:0/3449634549 >> 172.21.15.77:0/3449634549 conn(0x7f282c0d3ef0 msgr2=0x7f282c0d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 -- 172.21.15.77:0/3449634549 shutdown_connections 2024-04-25T01:23:04.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.035+0000 7f2833f42640 1 -- 172.21.15.77:0/3449634549 wait complete. 2024-04-25T01:23:04.027 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.457887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.029883+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:23:04.027 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:23:04.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.146+0000 7ff5f4872640 1 Processor -- start 2024-04-25T01:23:04.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.146+0000 7ff5f4872640 1 -- start start 2024-04-25T01:23:04.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.147+0000 7ff5f4872640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5f00b9a90 0x7ff5f00d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.147+0000 7ff5f4872640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff5f00d27b0 0x7ff5f00d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.147+0000 7ff5f4872640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5f00cea40 con 0x7ff5f00d23d0 2024-04-25T01:23:04.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.147+0000 7ff5f4872640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7ff5f00cece0 con 0x7ff5f00d27b0 2024-04-25T01:23:04.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.147+0000 7ff5f4872640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff5f00cefc0 con 0x7ff5f00b9a90 2024-04-25T01:23:04.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.147+0000 7ff5eed9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff5f00d27b0 0x7ff5f00d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.147+0000 7ff5eed9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff5f00d27b0 0x7ff5f00d0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44228/0 (socket says 172.21.15.77:44228) 2024-04-25T01:23:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.147+0000 7ff5eed9d640 1 -- 172.21.15.77:0/4163639379 learned_addr learned my addr 172.21.15.77:0/4163639379 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.148+0000 7ff5edd9b640 1 --2- 172.21.15.77:0/4163639379 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5f00b9a90 0x7ff5f00d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.148+0000 7ff5eed9d640 1 -- 172.21.15.77:0/4163639379 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5f00b9a90 msgr2=0x7ff5f00d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.148+0000 7ff5eed9d640 1 --2- 172.21.15.77:0/4163639379 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5f00b9a90 0x7ff5f00d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.148+0000 7ff5eed9d640 1 -- 172.21.15.77:0/4163639379 >> v1:172.21.15.121:6789/0 conn(0x7ff5f00d23d0 legacy=0x7ff5f00b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.148+0000 7ff5eed9d640 1 -- 172.21.15.77:0/4163639379 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f00cf2a0 con 0x7ff5f00d27b0 2024-04-25T01:23:04.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.148+0000 7ff5eed9d640 1 --2- 172.21.15.77:0/4163639379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f00d27b0 0x7ff5f00d0ee0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8009870 tx=0x7ff5e802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653a8e2139a9a89e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.148+0000 7ff5ed59a640 1 -- 172.21.15.77:0/4163639379 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8037070 con 0x7ff5f00d27b0 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.149+0000 7ff5ed59a640 1 -- 172.21.15.77:0/4163639379 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e8032440 con 0x7ff5f00d27b0 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.149+0000 7ff5ed59a640 1 -- 172.21.15.77:0/4163639379 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8032810 con 0x7ff5f00d27b0 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.149+0000 7ff5f4872640 1 -- 172.21.15.77:0/4163639379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f00d27b0 msgr2=0x7ff5f00d0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.149+0000 7ff5f4872640 1 --2- 172.21.15.77:0/4163639379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f00d27b0 0x7ff5f00d0ee0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8009870 tx=0x7ff5e802a2a0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.149+0000 7ff5f4872640 1 -- 172.21.15.77:0/4163639379 shutdown_connections 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.149+0000 7ff5f4872640 1 --2- 172.21.15.77:0/4163639379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f00d27b0 0x7ff5f00d0ee0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.149+0000 7ff5f4872640 1 --2- 172.21.15.77:0/4163639379 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5f00b9a90 0x7ff5f00d4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.149+0000 7ff5f4872640 1 -- 172.21.15.77:0/4163639379 >> 172.21.15.77:0/4163639379 conn(0x7ff5f00d3e30 msgr2=0x7ff5f00d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.149+0000 7ff5f4872640 1 -- 172.21.15.77:0/4163639379 shutdown_connections 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.149+0000 7ff5f4872640 1 -- 172.21.15.77:0/4163639379 wait complete. 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.150+0000 7ff5f4872640 1 Processor -- start 2024-04-25T01:23:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.150+0000 7ff5f4872640 1 -- start start 2024-04-25T01:23:04.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.151+0000 7ff5f4872640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5f00b9a90 0x7ff5f0152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.151+0000 7ff5f4872640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f00d23d0 0x7ff5f00c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.151+0000 7ff5f4872640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5f00d27b0 0x7ff5f006bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.151+0000 7ff5f4872640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f00cfd80 con 0x7ff5f00d23d0 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.151+0000 7ff5f4872640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f00cf860 con 0x7ff5f00d27b0 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.151+0000 7ff5f4872640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff5f00b6c80 con 0x7ff5f00b9a90 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.151+0000 7ff5edd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f00d23d0 0x7ff5f00c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.151+0000 7ff5edd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f00d23d0 0x7ff5f00c7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44232/0 (socket says 172.21.15.77:44232) 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.151+0000 7ff5edd9b640 1 -- 172.21.15.77:0/3019274398 learned_addr learned my addr 172.21.15.77:0/3019274398 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.151+0000 7ff5ee59c640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5f00b9a90 0x7ff5f0152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5eed9d640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5f00d27b0 0x7ff5f006bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5edd9b640 1 -- 172.21.15.77:0/3019274398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5f00b9a90 msgr2=0x7ff5f0152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5edd9b640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5f00b9a90 0x7ff5f0152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5edd9b640 1 -- 172.21.15.77:0/3019274398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5f00d27b0 msgr2=0x7ff5f006bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5edd9b640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5f00d27b0 0x7ff5f006bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5edd9b640 1 -- 172.21.15.77:0/3019274398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f00b1340 con 0x7ff5f00d23d0 2024-04-25T01:23:04.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5eed9d640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5f00d27b0 0x7ff5f006bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:04.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5edd9b640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f00d23d0 0x7ff5f00c7960 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400c7f0 tx=0x7ff5e400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5deffd640 1 -- 172.21.15.77:0/3019274398 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4015070 con 0x7ff5f00d23d0 2024-04-25T01:23:04.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5deffd640 1 -- 172.21.15.77:0/3019274398 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e4004240 con 0x7ff5f00d23d0 2024-04-25T01:23:04.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5f4872640 1 -- 172.21.15.77:0/3019274398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f00c83b0 con 0x7ff5f00d23d0 2024-04-25T01:23:04.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5deffd640 1 -- 172.21.15.77:0/3019274398 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4007620 con 0x7ff5f00d23d0 2024-04-25T01:23:04.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.152+0000 7ff5f4872640 1 -- 172.21.15.77:0/3019274398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f00c8970 con 0x7ff5f00d23d0 2024-04-25T01:23:04.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.153+0000 7ff5f4872640 1 -- 172.21.15.77:0/3019274398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5f006bad0 con 0x7ff5f00d23d0 2024-04-25T01:23:04.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.153+0000 7ff5deffd640 1 -- 172.21.15.77:0/3019274398 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff5e40077c0 con 0x7ff5f00d23d0 2024-04-25T01:23:04.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.154+0000 7ff5deffd640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff5d006be60 0x7ff5d006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.154+0000 7ff5deffd640 1 -- 172.21.15.77:0/3019274398 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7ff5e408ada0 con 0x7ff5f00d23d0 2024-04-25T01:23:04.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.154+0000 7ff5ee59c640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff5d006be60 0x7ff5d006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.154+0000 7ff5ee59c640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff5d006be60 0x7ff5d006e320 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff5f01611f0 tx=0x7ff5e0003d90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.158+0000 7ff5deffd640 1 -- 172.21.15.77:0/3019274398 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff5f006bad0 con 0x7ff5f00d23d0 2024-04-25T01:23:04.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.330+0000 7ff5f4872640 1 -- 172.21.15.77:0/3019274398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff5f008df50 con 0x7ff5f00d23d0 2024-04-25T01:23:04.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.330+0000 7ff5deffd640 1 -- 172.21.15.77:0/3019274398 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) ==== 93+0+10 (secure 0 0 0) 0x7ff5e4058f80 con 0x7ff5f00d23d0 2024-04-25T01:23:04.310 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:23:04.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 -- 172.21.15.77:0/3019274398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff5d006be60 msgr2=0x7ff5d006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff5d006be60 0x7ff5d006e320 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff5f01611f0 tx=0x7ff5e0003d90 comp rx=0 tx=0).stop 2024-04-25T01:23:04.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 -- 172.21.15.77:0/3019274398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f00d23d0 msgr2=0x7ff5f00c7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f00d23d0 0x7ff5f00c7960 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400c7f0 tx=0x7ff5e400ccb0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 -- 172.21.15.77:0/3019274398 shutdown_connections 2024-04-25T01:23:04.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff5d006be60 0x7ff5d006e320 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f00d23d0 0x7ff5f00c7960 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5f00d27b0 0x7ff5f006bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 --2- 172.21.15.77:0/3019274398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5f00b9a90 0x7ff5f0152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 -- 172.21.15.77:0/3019274398 >> 172.21.15.77:0/3019274398 conn(0x7ff5f00d3e30 msgr2=0x7ff5f00ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 -- 172.21.15.77:0/3019274398 shutdown_connections 2024-04-25T01:23:04.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.332+0000 7ff5f4872640 1 -- 172.21.15.77:0/3019274398 wait complete. 2024-04-25T01:23:04.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:23:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.442+0000 7fab1a0a5640 1 Processor -- start 2024-04-25T01:23:04.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.442+0000 7fab1a0a5640 1 -- start start 2024-04-25T01:23:04.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.442+0000 7fab1a0a5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fab14158a70 0x7fab140b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab1a0a5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140b37c0 0x7fab140b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab1a0a5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab1415c210 con 0x7fab1415b520 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab1a0a5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fab140b2ce0 con 0x7fab14158a70 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab1a0a5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fab140b2fc0 con 0x7fab140b37c0 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab13fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fab1415b520 0x7fab1415b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58538/0 (socket says 172.21.15.77:58538) 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab13fff640 1 -- 172.21.15.77:0/476228669 learned_addr learned my addr 172.21.15.77:0/476228669 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab12ffd640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140b37c0 0x7fab140b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab137fe640 1 --2- 172.21.15.77:0/476228669 >> v2:172.21.15.121:3300/0 conn(0x7fab14158a70 0x7fab140b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1718928638 0 0) 0x7fab1415c210 con 0x7fab1415b520 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faafc005730 con 0x7fab1415b520 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 101031576 0 0) 0x7faafc005730 con 0x7fab1415b520 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140b37c0 msgr2=0x7fab140b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140b37c0 0x7fab140b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 >> v2:172.21.15.121:3300/0 conn(0x7fab14158a70 msgr2=0x7fab140b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 --2- 172.21.15.77:0/476228669 >> v2:172.21.15.121:3300/0 conn(0x7fab14158a70 0x7fab140b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab140b32a0 con 0x7fab1415b520 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fab0c0019f0 con 0x7fab1415b520 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 >> v1:172.21.15.77:6789/0 conn(0x7fab1415b520 legacy=0x7fab1415b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140b37c0 0x7faafc005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 0x7faafc006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faafc006f50 0x7faafc0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fab140b2ce0 con 0x7faafc006140 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fab140b2fc0 con 0x7faafc006f50 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faafc00dbd0 con 0x7fab140b37c0 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.443+0000 7fab12ffd640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 0x7faafc006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab13fff640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faafc006f50 0x7faafc0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab137fe640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140b37c0 0x7faafc005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab12ffd640 1 -- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140b37c0 msgr2=0x7faafc005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab12ffd640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140b37c0 0x7faafc005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab12ffd640 1 -- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faafc006f50 msgr2=0x7faafc0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab12ffd640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faafc006f50 0x7faafc0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab12ffd640 1 -- 172.21.15.77:0/476228669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faafc005730 con 0x7faafc006140 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab137fe640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140b37c0 0x7faafc005bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab12ffd640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 0x7faafc006a10 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fab08009ac0 tx=0x7faafc007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e50cf505d73e895 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08036070 con 0x7faafc006140 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab08031400 con 0x7faafc006140 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.444+0000 7fab127fc640 1 -- 172.21.15.77:0/476228669 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08009690 con 0x7faafc006140 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.445+0000 7fab1a0a5640 1 -- 172.21.15.77:0/476228669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 msgr2=0x7faafc006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.445+0000 7fab1a0a5640 1 --2- 172.21.15.77:0/476228669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 0x7faafc006a10 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fab08009ac0 tx=0x7faafc007a40 comp rx=0 tx=0).stop 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.445+0000 7fab13fff640 1 -- 172.21.15.77:0/476228669 reap_dead start 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.445+0000 7fab1a0a5640 1 -- 172.21.15.77:0/476228669 shutdown_connections 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.445+0000 7fab1a0a5640 1 -- 172.21.15.77:0/476228669 >> 172.21.15.77:0/476228669 conn(0x7fab1406b720 msgr2=0x7fab140b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.445+0000 7fab1a0a5640 1 -- 172.21.15.77:0/476228669 shutdown_connections 2024-04-25T01:23:04.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.445+0000 7fab1a0a5640 1 -- 172.21.15.77:0/476228669 wait complete. 2024-04-25T01:23:04.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.446+0000 7fab1a0a5640 1 Processor -- start 2024-04-25T01:23:04.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.446+0000 7fab1a0a5640 1 -- start start 2024-04-25T01:23:04.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab1a0a5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 0x7fab140a9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab1a0a5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140aec00 0x7fab140af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab1a0a5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab140cdd90 0x7fab140ce240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab1a0a5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fab140b4780 con 0x7faafc006140 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab1a0a5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fab140b4600 con 0x7fab140cdd90 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab1a0a5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fab140b4900 con 0x7fab140aec00 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 0x7fab140a9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 0x7fab140a9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44256/0 (socket says 172.21.15.77:44256) 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab137fe640 1 -- 172.21.15.77:0/2791261287 learned_addr learned my addr 172.21.15.77:0/2791261287 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab12ffd640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140aec00 0x7fab140af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab13fff640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab140cdd90 0x7fab140ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab137fe640 1 -- 172.21.15.77:0/2791261287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140aec00 msgr2=0x7fab140af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab137fe640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140aec00 0x7fab140af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab137fe640 1 -- 172.21.15.77:0/2791261287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab140cdd90 msgr2=0x7fab140ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab137fe640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab140cdd90 0x7fab140ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.447+0000 7fab137fe640 1 -- 172.21.15.77:0/2791261287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab14130090 con 0x7faafc006140 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.448+0000 7fab137fe640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 0x7fab140a9740 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fab04011700 tx=0x7fab04011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.448+0000 7faafbfff640 1 -- 172.21.15.77:0/2791261287 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0400f9c0 con 0x7faafc006140 2024-04-25T01:23:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.448+0000 7faafbfff640 1 -- 172.21.15.77:0/2791261287 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab0400fb60 con 0x7faafc006140 2024-04-25T01:23:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.448+0000 7fab1a0a5640 1 -- 172.21.15.77:0/2791261287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab14130370 con 0x7faafc006140 2024-04-25T01:23:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.448+0000 7faafbfff640 1 -- 172.21.15.77:0/2791261287 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab04015460 con 0x7faafc006140 2024-04-25T01:23:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.448+0000 7fab1a0a5640 1 -- 172.21.15.77:0/2791261287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab14130930 con 0x7faafc006140 2024-04-25T01:23:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.449+0000 7fab1a0a5640 1 -- 172.21.15.77:0/2791261287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab14068fb0 con 0x7faafc006140 2024-04-25T01:23:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.449+0000 7faafbfff640 1 -- 172.21.15.77:0/2791261287 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fab04015600 con 0x7faafc006140 2024-04-25T01:23:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.450+0000 7faafbfff640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fab0006be60 0x7fab0006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.450+0000 7fab12ffd640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fab0006be60 0x7fab0006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.450+0000 7faafbfff640 1 -- 172.21.15.77:0/2791261287 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fab0408b9a0 con 0x7faafc006140 2024-04-25T01:23:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.450+0000 7fab12ffd640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fab0006be60 0x7fab0006e320 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fab08033000 tx=0x7fab08008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.454+0000 7faafbfff640 1 -- 172.21.15.77:0/2791261287 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fab04019030 con 0x7faafc006140 2024-04-25T01:23:04.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.626+0000 7fab1a0a5640 1 -- 172.21.15.77:0/2791261287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fab140b3c70 con 0x7faafc006140 2024-04-25T01:23:04.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.626+0000 7faafbfff640 1 -- 172.21.15.77:0/2791261287 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) ==== 104+0+11 (secure 0 0 0) 0x7fab0405a1c0 con 0x7faafc006140 2024-04-25T01:23:04.607 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:23:04.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.628+0000 7fab1a0a5640 1 -- 172.21.15.77:0/2791261287 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fab0006be60 msgr2=0x7fab0006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.628+0000 7fab1a0a5640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fab0006be60 0x7fab0006e320 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fab08033000 tx=0x7fab08008030 comp rx=0 tx=0).stop 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.628+0000 7fab1a0a5640 1 -- 172.21.15.77:0/2791261287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 msgr2=0x7fab140a9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.628+0000 7fab1a0a5640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 0x7fab140a9740 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fab04011700 tx=0x7fab04011bc0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.628+0000 7fab1a0a5640 1 -- 172.21.15.77:0/2791261287 shutdown_connections 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.628+0000 7fab1a0a5640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fab0006be60 0x7fab0006e320 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.628+0000 7fab1a0a5640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab140cdd90 0x7fab140ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.628+0000 7fab1a0a5640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab140aec00 0x7fab140af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.628+0000 7fab1a0a5640 1 --2- 172.21.15.77:0/2791261287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faafc006140 0x7fab140a9740 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.628+0000 7fab1a0a5640 1 -- 172.21.15.77:0/2791261287 >> 172.21.15.77:0/2791261287 conn(0x7fab1406b720 msgr2=0x7fab14159860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.629+0000 7fab1a0a5640 1 -- 172.21.15.77:0/2791261287 shutdown_connections 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.629+0000 7fab1a0a5640 1 -- 172.21.15.77:0/2791261287 wait complete. 2024-04-25T01:23:04.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:23:04.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.785+0000 7f3c62ae1640 1 Processor -- start 2024-04-25T01:23:04.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.785+0000 7f3c62ae1640 1 -- start start 2024-04-25T01:23:04.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c62ae1640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3c5c0b47d0 0x7f3c5c15b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c62ae1640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5c15c210 con 0x7f3c5c0b37c0 2024-04-25T01:23:04.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c62ae1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5c0b2ce0 con 0x7f3c5c158a70 2024-04-25T01:23:04.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c62ae1640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3c5c0b2fc0 con 0x7f3c5c0b47d0 2024-04-25T01:23:04.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c6107e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3c5c0b37c0 0x7f3c5c15ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58552/0 (socket says 172.21.15.77:58552) 2024-04-25T01:23:04.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c6107e640 1 -- 172.21.15.77:0/2286139874 learned_addr learned my addr 172.21.15.77:0/2286139874 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:04.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3132548741 0 0) 0x7f3c5c15c210 con 0x7f3c5c0b37c0 2024-04-25T01:23:04.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c4c003520 con 0x7f3c5c0b37c0 2024-04-25T01:23:04.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5bfff640 1 --2- 172.21.15.77:0/2286139874 >> v2:172.21.15.121:3300/0 conn(0x7f3c5c0b47d0 0x7f3c5c15b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3320650030 0 0) 0x7f3c4c003520 con 0x7f3c5c0b37c0 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 >> v2:172.21.15.121:3300/0 conn(0x7f3c5c0b47d0 msgr2=0x7f3c5c15b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 --2- 172.21.15.77:0/2286139874 >> v2:172.21.15.121:3300/0 conn(0x7f3c5c0b47d0 0x7f3c5c15b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 >> v1:172.21.15.121:6789/0 conn(0x7f3c5c158a70 legacy=0x7f3c5c0b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5c0b32a0 con 0x7f3c5c0b37c0 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f3c540019f0 con 0x7f3c5c0b37c0 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5bfff640 1 --2- 172.21.15.77:0/2286139874 >> v2:172.21.15.121:3300/0 conn(0x7f3c5c0b47d0 0x7f3c5c15b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 >> v1:172.21.15.77:6789/0 conn(0x7f3c5c0b37c0 legacy=0x7f3c5c15ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 --2- 172.21.15.77:0/2286139874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c4c0039b0 0x7f3c4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 --2- 172.21.15.77:0/2286139874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c4c004800 0x7f3c4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 --2- 172.21.15.77:0/2286139874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4c0051c0 0x7f3c4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c0b2fc0 con 0x7f3c4c0051c0 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4c00c110 con 0x7f3c4c004800 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3c4c00e520 con 0x7f3c4c0039b0 2024-04-25T01:23:04.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.786+0000 7f3c6107e640 1 --2- 172.21.15.77:0/2286139874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4c0051c0 0x7f3c4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.787+0000 7f3c5bfff640 1 --2- 172.21.15.77:0/2286139874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c4c004800 0x7f3c4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.787+0000 7f3c6087d640 1 --2- 172.21.15.77:0/2286139874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c4c0039b0 0x7f3c4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.787+0000 7f3c6107e640 1 -- 172.21.15.77:0/2286139874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c4c0039b0 msgr2=0x7f3c4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.787+0000 7f3c6107e640 1 --2- 172.21.15.77:0/2286139874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c4c0039b0 0x7f3c4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.787+0000 7f3c6107e640 1 -- 172.21.15.77:0/2286139874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c4c004800 msgr2=0x7f3c4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.787+0000 7f3c6107e640 1 --2- 172.21.15.77:0/2286139874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c4c004800 0x7f3c4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.787+0000 7f3c6107e640 1 -- 172.21.15.77:0/2286139874 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4c003520 con 0x7f3c4c0051c0 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.787+0000 7f3c6107e640 1 -- 172.21.15.77:0/2286139874 reap_dead start 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.787+0000 7f3c6107e640 1 --2- 172.21.15.77:0/2286139874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4c0051c0 0x7f3c4c00b9c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400a840 tx=0x7f3c4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da37924503ed0745 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.787+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5400a460 con 0x7f3c4c0051c0 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.788+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c5402f070 con 0x7f3c4c0051c0 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.788+0000 7f3c5b7fe640 1 -- 172.21.15.77:0/2286139874 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5402bcb0 con 0x7f3c4c0051c0 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.788+0000 7f3c62ae1640 1 -- 172.21.15.77:0/2286139874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4c0051c0 msgr2=0x7f3c4c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.788+0000 7f3c62ae1640 1 --2- 172.21.15.77:0/2286139874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4c0051c0 0x7f3c4c00b9c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400a840 tx=0x7f3c4c005830 comp rx=0 tx=0).stop 2024-04-25T01:23:04.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.788+0000 7f3c62ae1640 1 -- 172.21.15.77:0/2286139874 shutdown_connections 2024-04-25T01:23:04.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.788+0000 7f3c62ae1640 1 --2- 172.21.15.77:0/2286139874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4c0051c0 0x7f3c4c00b9c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.788+0000 7f3c62ae1640 1 -- 172.21.15.77:0/2286139874 >> 172.21.15.77:0/2286139874 conn(0x7f3c5c06b720 msgr2=0x7f3c5c0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.788+0000 7f3c62ae1640 1 -- 172.21.15.77:0/2286139874 shutdown_connections 2024-04-25T01:23:04.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.788+0000 7f3c62ae1640 1 -- 172.21.15.77:0/2286139874 wait complete. 2024-04-25T01:23:04.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.789+0000 7f3c62ae1640 1 Processor -- start 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.789+0000 7f3c62ae1640 1 -- start start 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c62ae1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c4c0051c0 0x7f3c5c157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c62ae1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c5c0c8ec0 0x7f3c5c0c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c62ae1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c5c0cef20 0x7f3c5c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c62ae1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c0b4240 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c62ae1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c0b40c0 con 0x7f3c4c0051c0 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c62ae1640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3c5c0b43c0 con 0x7f3c5c0cef20 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c5bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c5c0c8ec0 0x7f3c5c0c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c5bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c5c0c8ec0 0x7f3c5c0c9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44268/0 (socket says 172.21.15.77:44268) 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c5bfff640 1 -- 172.21.15.77:0/1166192570 learned_addr learned my addr 172.21.15.77:0/1166192570 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c6107e640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c5c0cef20 0x7f3c5c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c6087d640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c4c0051c0 0x7f3c5c157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.790+0000 7f3c5bfff640 1 -- 172.21.15.77:0/1166192570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c5c0cef20 msgr2=0x7f3c5c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c5bfff640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c5c0cef20 0x7f3c5c0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c5bfff640 1 -- 172.21.15.77:0/1166192570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c4c0051c0 msgr2=0x7f3c5c157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c5bfff640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c4c0051c0 0x7f3c5c157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c5bfff640 1 -- 172.21.15.77:0/1166192570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5c0cde80 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c6107e640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c5c0cef20 0x7f3c5c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c6087d640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c4c0051c0 0x7f3c5c157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c5bfff640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c5c0c8ec0 0x7f3c5c0c9340 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3c480115c0 tx=0x7f3c48011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c597fa640 1 -- 172.21.15.77:0/1166192570 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c48019070 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c597fa640 1 -- 172.21.15.77:0/1166192570 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c48004220 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c62ae1640 1 -- 172.21.15.77:0/1166192570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5c0ce160 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c597fa640 1 -- 172.21.15.77:0/1166192570 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4800f040 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.791+0000 7f3c62ae1640 1 -- 172.21.15.77:0/1166192570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c5c0ce720 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.792+0000 7f3c62ae1640 1 -- 172.21.15.77:0/1166192570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c5c158280 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.792+0000 7f3c597fa640 1 -- 172.21.15.77:0/1166192570 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3c48015670 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.793+0000 7f3c597fa640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3c4006be30 0x7f3c4006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.793+0000 7f3c6087d640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3c4006be30 0x7f3c4006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.793+0000 7f3c597fa640 1 -- 172.21.15.77:0/1166192570 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f3c4808b250 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.793+0000 7f3c6087d640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3c4006be30 0x7f3c4006e2f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3c50005580 tx=0x7f3c50009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.797+0000 7f3c597fa640 1 -- 172.21.15.77:0/1166192570 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3c5c158280 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.969+0000 7f3c62ae1640 1 -- 172.21.15.77:0/1166192570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3c5c068fb0 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.970+0000 7f3c597fa640 1 -- 172.21.15.77:0/1166192570 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) ==== 100+0+11 (secure 0 0 0) 0x7f3c48059a70 con 0x7f3c5c0c8ec0 2024-04-25T01:23:04.951 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:23:04.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 -- 172.21.15.77:0/1166192570 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3c4006be30 msgr2=0x7f3c4006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3c4006be30 0x7f3c4006e2f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3c50005580 tx=0x7f3c50009040 comp rx=0 tx=0).stop 2024-04-25T01:23:04.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 -- 172.21.15.77:0/1166192570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c5c0c8ec0 msgr2=0x7f3c5c0c9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c5c0c8ec0 0x7f3c5c0c9340 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3c480115c0 tx=0x7f3c48011a80 comp rx=0 tx=0).stop 2024-04-25T01:23:04.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 -- 172.21.15.77:0/1166192570 shutdown_connections 2024-04-25T01:23:04.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3c4006be30 0x7f3c4006e2f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c5c0c8ec0 0x7f3c5c0c9340 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c5c0cef20 0x7f3c5c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 --2- 172.21.15.77:0/1166192570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c4c0051c0 0x7f3c5c157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 -- 172.21.15.77:0/1166192570 >> 172.21.15.77:0/1166192570 conn(0x7f3c5c06b720 msgr2=0x7f3c5c159f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 -- 172.21.15.77:0/1166192570 shutdown_connections 2024-04-25T01:23:04.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:04.972+0000 7f3c62ae1640 1 -- 172.21.15.77:0/1166192570 wait complete. 2024-04-25T01:23:04.963 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:23:05.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.080+0000 7f4af8129640 1 Processor -- start 2024-04-25T01:23:05.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.081+0000 7f4af8129640 1 -- start start 2024-04-25T01:23:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.081+0000 7f4af8129640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4af0161730 0x7f4af00d1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.081+0000 7f4af8129640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4af00cf650 0x7f4af00d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.081+0000 7f4af8129640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4af00d0ba0 0x7f4af00d1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.081+0000 7f4af8129640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f4af00d1010 con 0x7f4af0161730 2024-04-25T01:23:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.081+0000 7f4af8129640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4af00cfbc0 con 0x7f4af00cf650 2024-04-25T01:23:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.081+0000 7f4af8129640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4af00cfea0 con 0x7f4af00d0ba0 2024-04-25T01:23:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.082+0000 7f4af5ec5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4af0161730 0x7f4af00d1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.082+0000 7f4af5ec5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4af0161730 0x7f4af00d1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44270/0 (socket says 172.21.15.77:44270) 2024-04-25T01:23:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.082+0000 7f4af5ec5640 1 -- 172.21.15.77:0/2574568095 learned_addr learned my addr 172.21.15.77:0/2574568095 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.082+0000 7f4af66c6640 1 --2- 172.21.15.77:0/2574568095 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4af00d0ba0 0x7f4af00d1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.082+0000 7f4af56c4640 1 --2- 172.21.15.77:0/2574568095 >> v2:172.21.15.121:3300/0 conn(0x7f4af00cf650 0x7f4af00d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.082+0000 7f4af5ec5640 1 -- 172.21.15.77:0/2574568095 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4af00d0ba0 msgr2=0x7f4af00d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.082+0000 7f4af5ec5640 1 --2- 172.21.15.77:0/2574568095 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4af00d0ba0 0x7f4af00d1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.082+0000 7f4af5ec5640 1 -- 172.21.15.77:0/2574568095 >> v2:172.21.15.121:3300/0 conn(0x7f4af00cf650 msgr2=0x7f4af00d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.082+0000 7f4af5ec5640 1 --2- 172.21.15.77:0/2574568095 >> v2:172.21.15.121:3300/0 conn(0x7f4af00cf650 0x7f4af00d0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.082+0000 7f4af5ec5640 1 -- 172.21.15.77:0/2574568095 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af00b34f0 con 0x7f4af0161730 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.082+0000 7f4af5ec5640 1 --2- 172.21.15.77:0/2574568095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af0161730 0x7f4af00d1200 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4af0067d50 tx=0x7f4ae802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6168f52cdf6d3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af4ec3640 1 -- 172.21.15.77:0/2574568095 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8037070 con 0x7f4af0161730 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af4ec3640 1 -- 172.21.15.77:0/2574568095 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae8032440 con 0x7f4af0161730 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af4ec3640 1 -- 172.21.15.77:0/2574568095 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8032810 con 0x7f4af0161730 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af8129640 1 -- 172.21.15.77:0/2574568095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af0161730 msgr2=0x7f4af00d1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af8129640 1 --2- 172.21.15.77:0/2574568095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af0161730 0x7f4af00d1200 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4af0067d50 tx=0x7f4ae802a1e0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af8129640 1 -- 172.21.15.77:0/2574568095 shutdown_connections 2024-04-25T01:23:05.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af8129640 1 --2- 172.21.15.77:0/2574568095 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4af00d0ba0 0x7f4af00d1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af8129640 1 --2- 172.21.15.77:0/2574568095 >> v2:172.21.15.121:3300/0 conn(0x7f4af00cf650 0x7f4af00d0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af8129640 1 --2- 172.21.15.77:0/2574568095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af0161730 0x7f4af00d1200 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af8129640 1 -- 172.21.15.77:0/2574568095 >> 172.21.15.77:0/2574568095 conn(0x7f4af00d3f30 msgr2=0x7f4af00d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:05.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af8129640 1 -- 172.21.15.77:0/2574568095 shutdown_connections 2024-04-25T01:23:05.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.083+0000 7f4af8129640 1 -- 172.21.15.77:0/2574568095 wait complete. 2024-04-25T01:23:05.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.084+0000 7f4af8129640 1 Processor -- start 2024-04-25T01:23:05.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.084+0000 7f4af8129640 1 -- start start 2024-04-25T01:23:05.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.085+0000 7f4af8129640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af00cf650 0x7f4af00c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.085+0000 7f4af8129640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af00d0ba0 0x7f4af014f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.085+0000 7f4af8129640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4af0161730 0x7f4af00b9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.085+0000 7f4af8129640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4af00d1740 con 0x7f4af00cf650 2024-04-25T01:23:05.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.085+0000 7f4af8129640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4af00b3ab0 con 0x7f4af00d0ba0 2024-04-25T01:23:05.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.085+0000 7f4af8129640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4af00d18c0 con 0x7f4af0161730 2024-04-25T01:23:05.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.085+0000 7f4af5ec5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af00cf650 0x7f4af00c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.085+0000 7f4af5ec5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af00cf650 0x7f4af00c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44278/0 (socket says 172.21.15.77:44278) 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.085+0000 7f4af5ec5640 1 -- 172.21.15.77:0/1387898627 learned_addr learned my addr 172.21.15.77:0/1387898627 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.085+0000 7f4af66c6640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4af0161730 0x7f4af00b9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4af56c4640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af00d0ba0 0x7f4af014f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4af5ec5640 1 -- 172.21.15.77:0/1387898627 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4af0161730 msgr2=0x7f4af00b9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4af5ec5640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4af0161730 0x7f4af00b9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4af5ec5640 1 -- 172.21.15.77:0/1387898627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af00d0ba0 msgr2=0x7f4af014f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4af5ec5640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af00d0ba0 0x7f4af014f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4af5ec5640 1 -- 172.21.15.77:0/1387898627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af00c7f00 con 0x7f4af00cf650 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4af66c6640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4af0161730 0x7f4af00b9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4af56c4640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af00d0ba0 0x7f4af014f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4af5ec5640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af00cf650 0x7f4af00c79c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4ae802a220 tx=0x7f4ae80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4ae67fc640 1 -- 172.21.15.77:0/1387898627 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8041020 con 0x7f4af00cf650 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4ae67fc640 1 -- 172.21.15.77:0/1387898627 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae80306c0 con 0x7f4af00cf650 2024-04-25T01:23:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4af8129640 1 -- 172.21.15.77:0/1387898627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af00cc270 con 0x7f4af00cf650 2024-04-25T01:23:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4ae67fc640 1 -- 172.21.15.77:0/1387898627 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8037040 con 0x7f4af00cf650 2024-04-25T01:23:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.086+0000 7f4af8129640 1 -- 172.21.15.77:0/1387898627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af00cc830 con 0x7f4af00cf650 2024-04-25T01:23:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.087+0000 7f4af8129640 1 -- 172.21.15.77:0/1387898627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af0068fb0 con 0x7f4af00cf650 2024-04-25T01:23:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.087+0000 7f4ae67fc640 1 -- 172.21.15.77:0/1387898627 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4ae8044020 con 0x7f4af00cf650 2024-04-25T01:23:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.088+0000 7f4ae67fc640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4ad406be30 0x7f4ad406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.088+0000 7f4ae67fc640 1 -- 172.21.15.77:0/1387898627 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f4ae80ab830 con 0x7f4af00cf650 2024-04-25T01:23:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.088+0000 7f4af56c4640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4ad406be30 0x7f4ad406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.088+0000 7f4af56c4640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4ad406be30 0x7f4ad406e2f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0006fd0 tx=0x7f4ae0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.092+0000 7f4ae67fc640 1 -- 172.21.15.77:0/1387898627 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4ae804b030 con 0x7f4af00cf650 2024-04-25T01:23:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.279+0000 7f4af8129640 1 -- 172.21.15.77:0/1387898627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4af00b2900 con 0x7f4af00cf650 2024-04-25T01:23:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.279+0000 7f4ae67fc640 1 -- 172.21.15.77:0/1387898627 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) ==== 80+0+0 (secure 0 0 0) 0x7f4ae807a050 con 0x7f4af00cf650 2024-04-25T01:23:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-04-25T01:23:05.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 -- 172.21.15.77:0/1387898627 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4ad406be30 msgr2=0x7f4ad406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4ad406be30 0x7f4ad406e2f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0006fd0 tx=0x7f4ae0008040 comp rx=0 tx=0).stop 2024-04-25T01:23:05.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 -- 172.21.15.77:0/1387898627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af00cf650 msgr2=0x7f4af00c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af00cf650 0x7f4af00c79c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4ae802a220 tx=0x7f4ae80046c0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 -- 172.21.15.77:0/1387898627 shutdown_connections 2024-04-25T01:23:05.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4ad406be30 0x7f4ad406e2f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4af0161730 0x7f4af00b9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af00d0ba0 0x7f4af014f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 --2- 172.21.15.77:0/1387898627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af00cf650 0x7f4af00c79c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 -- 172.21.15.77:0/1387898627 >> 172.21.15.77:0/1387898627 conn(0x7f4af00d3f30 msgr2=0x7f4af00b3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:05.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 -- 172.21.15.77:0/1387898627 shutdown_connections 2024-04-25T01:23:05.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.281+0000 7f4af8129640 1 -- 172.21.15.77:0/1387898627 wait complete. 2024-04-25T01:23:05.272 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T01:23:05.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:05.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:05.316 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:23:05.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:05.316 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T01:23:05.371 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:05.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.527+0000 7f8bf2288640 1 Processor -- start 2024-04-25T01:23:05.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.527+0000 7f8bf2288640 1 -- start start 2024-04-25T01:23:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.528+0000 7f8bf2288640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8bec0cfaf0 0x7f8bec0d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.528+0000 7f8bf2288640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bec0d1e90 con 0x7f8bec0d4a10 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.528+0000 7f8bf2288640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bec0b54a0 con 0x7f8bec15e4c0 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.528+0000 7f8bf2288640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8bec0d2130 con 0x7f8bec0cfaf0 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.528+0000 7f8beb7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8bec0d4a10 0x7f8bec0b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58554/0 (socket says 172.21.15.77:58554) 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.528+0000 7f8beb7fe640 1 -- 172.21.15.77:0/842240952 learned_addr learned my addr 172.21.15.77:0/842240952 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8bf0825640 1 --2- 172.21.15.77:0/842240952 >> v2:172.21.15.121:3300/0 conn(0x7f8bec0cfaf0 0x7f8bec0d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 900633112 0 0) 0x7f8bec0d1e90 con 0x7f8bec0d4a10 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bd4003520 con 0x7f8bec0d4a10 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4098150940 0 0) 0x7f8bd4003520 con 0x7f8bec0d4a10 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 >> v2:172.21.15.121:3300/0 conn(0x7f8bec0cfaf0 msgr2=0x7f8bec0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 --2- 172.21.15.77:0/842240952 >> v2:172.21.15.121:3300/0 conn(0x7f8bec0cfaf0 0x7f8bec0d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 >> v1:172.21.15.121:6789/0 conn(0x7f8bec15e4c0 legacy=0x7f8bec0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bec0b5780 con 0x7f8bec0d4a10 2024-04-25T01:23:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f8be0001670 con 0x7f8bec0d4a10 2024-04-25T01:23:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 >> v1:172.21.15.77:6789/0 conn(0x7f8bec0d4a10 legacy=0x7f8bec0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 --2- 172.21.15.77:0/842240952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bd40039b0 0x7f8bd4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 --2- 172.21.15.77:0/842240952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bd40043b0 0x7f8bd4004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 --2- 172.21.15.77:0/842240952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd4005120 0x7f8bd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec0d2130 con 0x7f8bd40039b0 2024-04-25T01:23:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd400c060 con 0x7f8bd4005120 2024-04-25T01:23:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8bd400c260 con 0x7f8bd40043b0 2024-04-25T01:23:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8bebfff640 1 --2- 172.21.15.77:0/842240952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bd40039b0 0x7f8bd4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8bf0825640 1 --2- 172.21.15.77:0/842240952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd4005120 0x7f8bd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8beb7fe640 1 --2- 172.21.15.77:0/842240952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bd40043b0 0x7f8bd4004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8bebfff640 1 -- 172.21.15.77:0/842240952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bd40043b0 msgr2=0x7f8bd4004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8bebfff640 1 --2- 172.21.15.77:0/842240952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bd40043b0 0x7f8bd4004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8bebfff640 1 -- 172.21.15.77:0/842240952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd4005120 msgr2=0x7f8bd400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8bebfff640 1 --2- 172.21.15.77:0/842240952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd4005120 0x7f8bd400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8bebfff640 1 -- 172.21.15.77:0/842240952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd4003520 con 0x7f8bd40039b0 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.529+0000 7f8bf0825640 1 -- 172.21.15.77:0/842240952 reap_dead start 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.530+0000 7f8bebfff640 1 --2- 172.21.15.77:0/842240952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bd40039b0 0x7f8bd4003d90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc001910 tx=0x7f8bd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12dad7f1efc396f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.530+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc037070 con 0x7f8bd40039b0 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.530+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bdc02ac40 con 0x7f8bd40039b0 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.530+0000 7f8beaffd640 1 -- 172.21.15.77:0/842240952 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc0014e0 con 0x7f8bd40039b0 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.531+0000 7f8bf2288640 1 -- 172.21.15.77:0/842240952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bd40039b0 msgr2=0x7f8bd4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.531+0000 7f8bf2288640 1 --2- 172.21.15.77:0/842240952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bd40039b0 0x7f8bd4003d90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc001910 tx=0x7f8bd4005830 comp rx=0 tx=0).stop 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.531+0000 7f8bf2288640 1 -- 172.21.15.77:0/842240952 shutdown_connections 2024-04-25T01:23:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.531+0000 7f8bf2288640 1 --2- 172.21.15.77:0/842240952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bd40039b0 0x7f8bd4003d90 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.531+0000 7f8bf2288640 1 -- 172.21.15.77:0/842240952 >> 172.21.15.77:0/842240952 conn(0x7f8bec0d3ef0 msgr2=0x7f8bec0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:05.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.531+0000 7f8bf2288640 1 -- 172.21.15.77:0/842240952 shutdown_connections 2024-04-25T01:23:05.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.531+0000 7f8bf2288640 1 -- 172.21.15.77:0/842240952 wait complete. 2024-04-25T01:23:05.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.532+0000 7f8bf2288640 1 Processor -- start 2024-04-25T01:23:05.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.532+0000 7f8bf2288640 1 -- start start 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf2288640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd40039b0 0x7f8bec152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf2288640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bec0ca570 0x7f8bec1f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf2288640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec0b8980 0x7f8bec0b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf2288640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec15ea20 con 0x7f8bec0b8980 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf2288640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec15e8a0 con 0x7f8bd40039b0 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf2288640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8bec0d2800 con 0x7f8bec0ca570 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf0825640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec0b8980 0x7f8bec0b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf0825640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec0b8980 0x7f8bec0b6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44300/0 (socket says 172.21.15.77:44300) 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf0825640 1 -- 172.21.15.77:0/2841844591 learned_addr learned my addr 172.21.15.77:0/2841844591 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bebfff640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd40039b0 0x7f8bec152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8beb7fe640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bec0ca570 0x7f8bec1f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf0825640 1 -- 172.21.15.77:0/2841844591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bec0ca570 msgr2=0x7f8bec1f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf0825640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bec0ca570 0x7f8bec1f4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf0825640 1 -- 172.21.15.77:0/2841844591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd40039b0 msgr2=0x7f8bec152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf0825640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd40039b0 0x7f8bec152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf0825640 1 -- 172.21.15.77:0/2841844591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bec0b9220 con 0x7f8bec0b8980 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf0825640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec0b8980 0x7f8bec0b6b10 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0c79f0 tx=0x7f8be400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8be8ff9640 1 -- 172.21.15.77:0/2841844591 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be400fde0 con 0x7f8bec0b8980 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8be8ff9640 1 -- 172.21.15.77:0/2841844591 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be401d070 con 0x7f8bec0b8980 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8bf2288640 1 -- 172.21.15.77:0/2841844591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bec0c7c70 con 0x7f8bec0b8980 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.533+0000 7f8be8ff9640 1 -- 172.21.15.77:0/2841844591 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4015670 con 0x7f8bec0b8980 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.534+0000 7f8bf2288640 1 -- 172.21.15.77:0/2841844591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bec0c8230 con 0x7f8bec0b8980 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.535+0000 7f8bf2288640 1 -- 172.21.15.77:0/2841844591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bec0c8430 con 0x7f8bec0b8980 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.535+0000 7f8be8ff9640 1 -- 172.21.15.77:0/2841844591 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8be4015810 con 0x7f8bec0b8980 2024-04-25T01:23:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.535+0000 7f8be8ff9640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8bd806be60 0x7f8bd806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.535+0000 7f8be8ff9640 1 -- 172.21.15.77:0/2841844591 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f8be408b980 con 0x7f8bec0b8980 2024-04-25T01:23:05.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.535+0000 7f8bebfff640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8bd806be60 0x7f8bd806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.536+0000 7f8bebfff640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8bd806be60 0x7f8bd806e320 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc034000 tx=0x7f8bdc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.539+0000 7f8be8ff9640 1 -- 172.21.15.77:0/2841844591 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8bec0c8430 con 0x7f8bec0b8980 2024-04-25T01:23:05.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.752+0000 7f8bf2288640 1 -- 172.21.15.77:0/2841844591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bec068fb0 con 0x7f8bec0b8980 2024-04-25T01:23:05.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.752+0000 7f8be8ff9640 1 -- 172.21.15.77:0/2841844591 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5187 (secure 0 0 0) 0x7f8be4010040 con 0x7f8bec0b8980 2024-04-25T01:23:05.733 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:05.733 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3139470828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3139470828},{"type":"v1","addr":"172.21.15.77:6835","nonce":3139470828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.066906+0000","modified":"2024-04-25T01:22:59.063205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5358":{"gid":5358,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2944320661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2944320661},{"type":"v1","addr":"172.21.15.77:6837","nonce":2944320661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5358,"qdb_cluster":[5358]},"id":2}]} 2024-04-25T01:23:05.733 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 16 2024-04-25T01:23:05.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 -- 172.21.15.77:0/2841844591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8bd806be60 msgr2=0x7f8bd806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8bd806be60 0x7f8bd806e320 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc034000 tx=0x7f8bdc02e000 comp rx=0 tx=0).stop 2024-04-25T01:23:05.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 -- 172.21.15.77:0/2841844591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec0b8980 msgr2=0x7f8bec0b6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec0b8980 0x7f8bec0b6b10 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0c79f0 tx=0x7f8be400f580 comp rx=0 tx=0).stop 2024-04-25T01:23:05.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 -- 172.21.15.77:0/2841844591 shutdown_connections 2024-04-25T01:23:05.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8bd806be60 0x7f8bd806e320 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec0b8980 0x7f8bec0b6b10 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bec0ca570 0x7f8bec1f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 --2- 172.21.15.77:0/2841844591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd40039b0 0x7f8bec152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 -- 172.21.15.77:0/2841844591 >> 172.21.15.77:0/2841844591 conn(0x7f8bec0d3ef0 msgr2=0x7f8bec15cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:05.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 -- 172.21.15.77:0/2841844591 shutdown_connections 2024-04-25T01:23:05.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.754+0000 7f8bf2288640 1 -- 172.21.15.77:0/2841844591 wait complete. 2024-04-25T01:23:05.746 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-04-25T01:23:05.867 INFO:teuthology.orchestra.run.smithi077.stdout:{"message":"","return_code":0} 2024-04-25T01:23:05.868 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-25T01:23:05.868 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 2:0 2024-04-25T01:23:05.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.984+0000 7fc4c8a2a640 1 Processor -- start 2024-04-25T01:23:05.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.984+0000 7fc4c8a2a640 1 -- start start 2024-04-25T01:23:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.985+0000 7fc4c8a2a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc4c415e4c0 0x7fc4c40d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.985+0000 7fc4c8a2a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4c40d4a10 0x7fc4c40b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.985+0000 7fc4c8a2a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4c40d2d40 0x7fc4c40b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.985+0000 7fc4c8a2a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fc4c40d1e90 con 0x7fc4c40d2d40 2024-04-25T01:23:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.985+0000 7fc4c8a2a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc4c40b54a0 con 0x7fc4c415e4c0 2024-04-25T01:23:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.985+0000 7fc4c8a2a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc4c40d2130 con 0x7fc4c40d4a10 2024-04-25T01:23:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c2d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4c40d2d40 0x7fc4c40b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c2d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4c40d2d40 0x7fc4c40b4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44312/0 (socket says 172.21.15.77:44312) 2024-04-25T01:23:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c2d9d640 1 -- 172.21.15.77:0/1711031049 learned_addr learned my addr 172.21.15.77:0/1711031049 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c1d9b640 1 --2- 172.21.15.77:0/1711031049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4c40d4a10 0x7fc4c40b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c259c640 1 --2- 172.21.15.77:0/1711031049 >> v2:172.21.15.121:3300/0 conn(0x7fc4c415e4c0 0x7fc4c40d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c2d9d640 1 -- 172.21.15.77:0/1711031049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4c40d4a10 msgr2=0x7fc4c40b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c2d9d640 1 --2- 172.21.15.77:0/1711031049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4c40d4a10 0x7fc4c40b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c2d9d640 1 -- 172.21.15.77:0/1711031049 >> v2:172.21.15.121:3300/0 conn(0x7fc4c415e4c0 msgr2=0x7fc4c40d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c2d9d640 1 --2- 172.21.15.77:0/1711031049 >> v2:172.21.15.121:3300/0 conn(0x7fc4c415e4c0 0x7fc4c40d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c2d9d640 1 -- 172.21.15.77:0/1711031049 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c40b5780 con 0x7fc4c40d2d40 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c1d9b640 1 --2- 172.21.15.77:0/1711031049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4c40d4a10 0x7fc4c40b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.986+0000 7fc4c259c640 1 --2- 172.21.15.77:0/1711031049 >> v2:172.21.15.121:3300/0 conn(0x7fc4c415e4c0 0x7fc4c40d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c2d9d640 1 --2- 172.21.15.77:0/1711031049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40d2d40 0x7fc4c40b4420 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4067960 tx=0x7fc4bc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b836870d28230f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c159a640 1 -- 172.21.15.77:0/1711031049 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc036070 con 0x7fc4c40d2d40 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c159a640 1 -- 172.21.15.77:0/1711031049 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4bc031600 con 0x7fc4c40d2d40 2024-04-25T01:23:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c159a640 1 -- 172.21.15.77:0/1711031049 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc0319d0 con 0x7fc4c40d2d40 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1711031049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40d2d40 msgr2=0x7fc4c40b4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c8a2a640 1 --2- 172.21.15.77:0/1711031049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40d2d40 0x7fc4c40b4420 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4067960 tx=0x7fc4bc02a610 comp rx=0 tx=0).stop 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1711031049 shutdown_connections 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c8a2a640 1 --2- 172.21.15.77:0/1711031049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40d2d40 0x7fc4c40b4420 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c8a2a640 1 --2- 172.21.15.77:0/1711031049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4c40d4a10 0x7fc4c40b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c8a2a640 1 --2- 172.21.15.77:0/1711031049 >> v2:172.21.15.121:3300/0 conn(0x7fc4c415e4c0 0x7fc4c40d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1711031049 >> 172.21.15.77:0/1711031049 conn(0x7fc4c40d3ef0 msgr2=0x7fc4c40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1711031049 shutdown_connections 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.987+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1711031049 wait complete. 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.989+0000 7fc4c8a2a640 1 Processor -- start 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.989+0000 7fc4c8a2a640 1 -- start start 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.989+0000 7fc4c8a2a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c40d2d40 0x7fc4c41f7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.989+0000 7fc4c8a2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40d4a10 0x7fc4c40c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.989+0000 7fc4c8a2a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4c415e4c0 0x7fc4c40b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.989+0000 7fc4c8a2a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c415ea20 con 0x7fc4c40d4a10 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.989+0000 7fc4c8a2a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c415e8a0 con 0x7fc4c40d2d40 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.989+0000 7fc4c8a2a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc4c40d0900 con 0x7fc4c415e4c0 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40d4a10 0x7fc4c40c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40d4a10 0x7fc4c40c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44322/0 (socket says 172.21.15.77:44322) 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c1d9b640 1 -- 172.21.15.77:0/1714840257 learned_addr learned my addr 172.21.15.77:0/1714840257 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c1d9b640 1 -- 172.21.15.77:0/1714840257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4c415e4c0 msgr2=0x7fc4c40b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c1d9b640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4c415e4c0 0x7fc4c40b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c1d9b640 1 -- 172.21.15.77:0/1714840257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c40d2d40 msgr2=0x7fc4c41f7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c259c640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c40d2d40 0x7fc4c41f7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c1d9b640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c40d2d40 0x7fc4c41f7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c1d9b640 1 -- 172.21.15.77:0/1714840257 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c41f4fd0 con 0x7fc4c40d4a10 2024-04-25T01:23:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c1d9b640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40d4a10 0x7fc4c40c9e30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00c7f0 tx=0x7fc4ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4baffd640 1 -- 172.21.15.77:0/1714840257 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac015070 con 0x7fc4c40d4a10 2024-04-25T01:23:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4baffd640 1 -- 172.21.15.77:0/1714840257 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4ac007820 con 0x7fc4c40d4a10 2024-04-25T01:23:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4baffd640 1 -- 172.21.15.77:0/1714840257 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac007bf0 con 0x7fc4c40d4a10 2024-04-25T01:23:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1714840257 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4c41f4210 con 0x7fc4c40d4a10 2024-04-25T01:23:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.990+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1714840257 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4c41f47d0 con 0x7fc4c40d4a10 2024-04-25T01:23:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.991+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1714840257 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4c4068fb0 con 0x7fc4c40d4a10 2024-04-25T01:23:05.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.991+0000 7fc4baffd640 1 -- 172.21.15.77:0/1714840257 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc4ac018070 con 0x7fc4c40d4a10 2024-04-25T01:23:05.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.992+0000 7fc4baffd640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc4a406be30 0x7fc4a406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.992+0000 7fc4baffd640 1 -- 172.21.15.77:0/1714840257 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fc4ac08b050 con 0x7fc4c40d4a10 2024-04-25T01:23:05.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.992+0000 7fc4c259c640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc4a406be30 0x7fc4a406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.992+0000 7fc4c259c640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc4a406be30 0x7fc4a406e2f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc4c415b350 tx=0x7fc4b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:05.996+0000 7fc4baffd640 1 -- 172.21.15.77:0/1714840257 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc4ac059900 con 0x7fc4c40d4a10 2024-04-25T01:23:06.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.238+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1714840257 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "2:0"} v 0) -- 0x7fc4c40d3c50 con 0x7fc4c40d4a10 2024-04-25T01:23:06.218 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-04-25T01:23:06.238+0000 7f46b1a53640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:23:06.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.240+0000 7fc4baffd640 1 -- 172.21.15.77:0/1714840257 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "2:0"}]=0 failed mds gid 5358 v17) ==== 97+0+0 (secure 0 0 0) 0x7fc4ac059200 con 0x7fc4c40d4a10 2024-04-25T01:23:06.220 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 5358 2024-04-25T01:23:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.242+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1714840257 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc4a406be30 msgr2=0x7fc4a406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.242+0000 7fc4c8a2a640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc4a406be30 0x7fc4a406e2f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc4c415b350 tx=0x7fc4b4008040 comp rx=0 tx=0).stop 2024-04-25T01:23:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.242+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1714840257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40d4a10 msgr2=0x7fc4c40c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.242+0000 7fc4c8a2a640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40d4a10 0x7fc4c40c9e30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00c7f0 tx=0x7fc4ac00ccb0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.243+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1714840257 shutdown_connections 2024-04-25T01:23:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.243+0000 7fc4c8a2a640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc4a406be30 0x7fc4a406e2f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.243+0000 7fc4c8a2a640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40d4a10 0x7fc4c40c9e30 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.243+0000 7fc4c8a2a640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4c415e4c0 0x7fc4c40b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.243+0000 7fc4c8a2a640 1 --2- 172.21.15.77:0/1714840257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c40d2d40 0x7fc4c41f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.243+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1714840257 >> 172.21.15.77:0/1714840257 conn(0x7fc4c40d3ef0 msgr2=0x7fc4c40b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:06.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.243+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1714840257 shutdown_connections 2024-04-25T01:23:06.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.243+0000 7fc4c8a2a640 1 -- 172.21.15.77:0/1714840257 wait complete. 2024-04-25T01:23:06.225 INFO:tasks.ceph.mds.a.smithi077.stderr: -3977> 2024-04-25T01:23:02.704+0000 7faad970f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:23:06.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:06.255 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-04-25T01:23:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.353+0000 7faf07f7e640 1 Processor -- start 2024-04-25T01:23:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.354+0000 7faf07f7e640 1 -- start start 2024-04-25T01:23:06.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.354+0000 7faf07f7e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf00154ac0 0x7faf0015b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:06.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.354+0000 7faf07f7e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7faf0006e550 0x7faf0006e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:06.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.354+0000 7faf07f7e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf00154ea0 con 0x7faf000cf2a0 2024-04-25T01:23:06.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.354+0000 7faf07f7e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7faf0006d540 con 0x7faf00154ac0 2024-04-25T01:23:06.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.354+0000 7faf07f7e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7faf0006eea0 con 0x7faf0006e550 2024-04-25T01:23:06.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.355+0000 7faf05d1a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf00154ac0 0x7faf0015b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:06.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.355+0000 7faf05519640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7faf000cf2a0 0x7faf000711f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58556/0 (socket says 172.21.15.77:58556) 2024-04-25T01:23:06.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.355+0000 7faf05519640 1 -- 172.21.15.77:0/885920100 learned_addr learned my addr 172.21.15.77:0/885920100 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:06.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.355+0000 7faf05d1a640 1 -- 172.21.15.77:0/885920100 >> v2:172.21.15.121:3300/0 conn(0x7faf0006e550 msgr2=0x7faf0006e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:06.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.355+0000 7faf0651b640 1 --2- 172.21.15.77:0/885920100 >> v2:172.21.15.121:3300/0 conn(0x7faf0006e550 0x7faf0006e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:06.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.355+0000 7faf05d1a640 1 --2- 172.21.15.77:0/885920100 >> v2:172.21.15.121:3300/0 conn(0x7faf0006e550 0x7faf0006e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.355+0000 7faf05d1a640 1 -- 172.21.15.77:0/885920100 >> v1:172.21.15.77:6789/0 conn(0x7faf000cf2a0 legacy=0x7faf000711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.355+0000 7faf05d1a640 1 -- 172.21.15.77:0/885920100 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0006d790 con 0x7faf00154ac0 2024-04-25T01:23:06.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.355+0000 7faf04d18640 1 -- 172.21.15.77:0/885920100 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2446561206 0 0) 0x7faf00154ea0 con 0x7faf000cf2a0 2024-04-25T01:23:06.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf05d1a640 1 --2- 172.21.15.77:0/885920100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf00154ac0 0x7faf0015b5f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7faf000cc2b0 tx=0x7faef802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e80fad871a5ff15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf04d18640 1 -- 172.21.15.77:0/885920100 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8007740 con 0x7faf00154ac0 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf04d18640 1 -- 172.21.15.77:0/885920100 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef80078e0 con 0x7faf00154ac0 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf04d18640 1 -- 172.21.15.77:0/885920100 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8007cb0 con 0x7faf00154ac0 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf07f7e640 1 -- 172.21.15.77:0/885920100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf00154ac0 msgr2=0x7faf0015b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf07f7e640 1 --2- 172.21.15.77:0/885920100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf00154ac0 0x7faf0015b5f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7faf000cc2b0 tx=0x7faef802a680 comp rx=0 tx=0).stop 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf07f7e640 1 -- 172.21.15.77:0/885920100 shutdown_connections 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf07f7e640 1 --2- 172.21.15.77:0/885920100 >> v2:172.21.15.121:3300/0 conn(0x7faf0006e550 0x7faf0006e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf07f7e640 1 --2- 172.21.15.77:0/885920100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf00154ac0 0x7faf0015b5f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf07f7e640 1 -- 172.21.15.77:0/885920100 >> 172.21.15.77:0/885920100 conn(0x7faf000b1820 msgr2=0x7faf000b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf07f7e640 1 -- 172.21.15.77:0/885920100 shutdown_connections 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.356+0000 7faf07f7e640 1 -- 172.21.15.77:0/885920100 wait complete. 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.357+0000 7faf07f7e640 1 Processor -- start 2024-04-25T01:23:06.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.357+0000 7faf07f7e640 1 -- start start 2024-04-25T01:23:06.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf07f7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0006e550 0x7faf000ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:06.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf07f7e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf000cf2a0 0x7faf0014d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:06.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf07f7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf00154ac0 0x7faf00159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:06.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf07f7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faf0006e030 con 0x7faf0006e550 2024-04-25T01:23:06.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf07f7e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7faf0006dd50 con 0x7faf000cf2a0 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf07f7e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faf00071900 con 0x7faf00154ac0 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf05d1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0006e550 0x7faf000ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf05d1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0006e550 0x7faf000ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44344/0 (socket says 172.21.15.77:44344) 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf05d1a640 1 -- 172.21.15.77:0/1247852318 learned_addr learned my addr 172.21.15.77:0/1247852318 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf05519640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf000cf2a0 0x7faf0014d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf05d1a640 1 -- 172.21.15.77:0/1247852318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf00154ac0 msgr2=0x7faf00159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf0651b640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf00154ac0 0x7faf00159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf05d1a640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf00154ac0 0x7faf00159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf05d1a640 1 -- 172.21.15.77:0/1247852318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf000cf2a0 msgr2=0x7faf0014d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf05d1a640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf000cf2a0 0x7faf0014d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf05d1a640 1 -- 172.21.15.77:0/1247852318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf00156e20 con 0x7faf0006e550 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf05519640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf000cf2a0 0x7faf0014d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.358+0000 7faf0651b640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf00154ac0 0x7faf00159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:06.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.359+0000 7faf05d1a640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0006e550 0x7faf000ad8e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7faef8009e80 tx=0x7faef8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.359+0000 7faef67fc640 1 -- 172.21.15.77:0/1247852318 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef803f070 con 0x7faf0006e550 2024-04-25T01:23:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.359+0000 7faef67fc640 1 -- 172.21.15.77:0/1247852318 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef8031c40 con 0x7faf0006e550 2024-04-25T01:23:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.359+0000 7faf07f7e640 1 -- 172.21.15.77:0/1247852318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf00157100 con 0x7faf0006e550 2024-04-25T01:23:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.359+0000 7faef67fc640 1 -- 172.21.15.77:0/1247852318 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8043410 con 0x7faf0006e550 2024-04-25T01:23:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.359+0000 7faf07f7e640 1 -- 172.21.15.77:0/1247852318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf00158950 con 0x7faf0006e550 2024-04-25T01:23:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.360+0000 7faf07f7e640 1 -- 172.21.15.77:0/1247852318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf00068fb0 con 0x7faf0006e550 2024-04-25T01:23:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.360+0000 7faef67fc640 1 -- 172.21.15.77:0/1247852318 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7faef80435b0 con 0x7faf0006e550 2024-04-25T01:23:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.360+0000 7faef67fc640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faee406be30 0x7faee406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.361+0000 7faef67fc640 1 -- 172.21.15.77:0/1247852318 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7faef80ac5a0 con 0x7faf0006e550 2024-04-25T01:23:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.361+0000 7faf05519640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faee406be30 0x7faee406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.361+0000 7faf05519640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faee406be30 0x7faee406e2f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faf00070e50 tx=0x7faef0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:06.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.365+0000 7faef67fc640 1 -- 172.21.15.77:0/1247852318 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7faef807ae00 con 0x7faf0006e550 2024-04-25T01:23:06.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.572+0000 7faf07f7e640 1 -- 172.21.15.77:0/1247852318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf000af530 con 0x7faf0006e550 2024-04-25T01:23:06.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.572+0000 7faef67fc640 1 -- 172.21.15.77:0/1247852318 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+4356 (secure 0 0 0) 0x7faef807a700 con 0x7faf0006e550 2024-04-25T01:23:06.553 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:06.553 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.066906+0000","modified":"2024-04-25T01:23:06.243271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5355":{"gid":5355,"name":"c","rank":0,"incarnation":18,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6835/3139470828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3139470828},{"type":"v1","addr":"172.21.15.77:6835","nonce":3139470828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T01:23:06.553 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 18 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 -- 172.21.15.77:0/1247852318 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faee406be30 msgr2=0x7faee406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faee406be30 0x7faee406e2f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faf00070e50 tx=0x7faef0009040 comp rx=0 tx=0).stop 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 -- 172.21.15.77:0/1247852318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0006e550 msgr2=0x7faf000ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0006e550 0x7faf000ad8e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7faef8009e80 tx=0x7faef8004a40 comp rx=0 tx=0).stop 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 -- 172.21.15.77:0/1247852318 shutdown_connections 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7faee406be30 0x7faee406e2f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf00154ac0 0x7faf00159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf000cf2a0 0x7faf0014d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 --2- 172.21.15.77:0/1247852318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0006e550 0x7faf000ad8e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 -- 172.21.15.77:0/1247852318 >> 172.21.15.77:0/1247852318 conn(0x7faf000b1820 msgr2=0x7faf000b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 -- 172.21.15.77:0/1247852318 shutdown_connections 2024-04-25T01:23:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:06.574+0000 7faf07f7e640 1 -- 172.21.15.77:0/1247852318 wait complete. 2024-04-25T01:23:06.968 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:22:57.066906+0000', 'modified': '2024-04-25T01:23:06.243271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5355}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5355': {'gid': 5355, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.77:6835/3139470828', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 3139470828}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 3139470828}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T01:23:06.968 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5355:up:replay 2024-04-25T01:23:07.969 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:08.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.097+0000 7f29e9328640 1 Processor -- start 2024-04-25T01:23:08.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.097+0000 7f29e9328640 1 -- start start 2024-04-25T01:23:08.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.098+0000 7f29e9328640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f29e40705c0 0x7f29e4166c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:08.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.098+0000 7f29e9328640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e40af340 con 0x7f29e40d28a0 2024-04-25T01:23:08.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.098+0000 7f29e9328640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e40af620 con 0x7f29e415f630 2024-04-25T01:23:08.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.098+0000 7f29e9328640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f29e40af100 con 0x7f29e40705c0 2024-04-25T01:23:08.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.098+0000 7f29e37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f29e40705c0 0x7f29e4166c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:08.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.098+0000 7f29e27fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f29e40d28a0 0x7f29e406eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44854/0 (socket says 172.21.15.77:44854) 2024-04-25T01:23:08.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.098+0000 7f29e27fc640 1 -- 172.21.15.77:0/110756598 learned_addr learned my addr 172.21.15.77:0/110756598 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:08.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.099+0000 7f29e37fe640 1 -- 172.21.15.77:0/110756598 >> v1:172.21.15.121:6789/0 conn(0x7f29e415f630 legacy=0x7f29e40ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:08.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.099+0000 7f29e37fe640 1 -- 172.21.15.77:0/110756598 >> v1:172.21.15.77:6789/0 conn(0x7f29e40d28a0 legacy=0x7f29e406eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:08.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.099+0000 7f29e37fe640 1 -- 172.21.15.77:0/110756598 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e40af900 con 0x7f29e40705c0 2024-04-25T01:23:08.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.099+0000 7f29e1ffb640 1 -- 172.21.15.77:0/110756598 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3328668709 0 0) 0x7f29e40af340 con 0x7f29e40d28a0 2024-04-25T01:23:08.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.099+0000 7f29e37fe640 1 --2- 172.21.15.77:0/110756598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e40705c0 0x7f29e4166c30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0097a0 tx=0x7f29dc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852a1f530c36db21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:08.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.099+0000 7f29e1ffb640 1 -- 172.21.15.77:0/110756598 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc037070 con 0x7f29e40705c0 2024-04-25T01:23:08.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.099+0000 7f29e1ffb640 1 -- 172.21.15.77:0/110756598 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29dc032440 con 0x7f29e40705c0 2024-04-25T01:23:08.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.099+0000 7f29e1ffb640 1 -- 172.21.15.77:0/110756598 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc032810 con 0x7f29e40705c0 2024-04-25T01:23:08.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.100+0000 7f29e9328640 1 -- 172.21.15.77:0/110756598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e40705c0 msgr2=0x7f29e4166c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:08.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.100+0000 7f29e9328640 1 --2- 172.21.15.77:0/110756598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e40705c0 0x7f29e4166c30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0097a0 tx=0x7f29dc02a1e0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.100+0000 7f29e9328640 1 -- 172.21.15.77:0/110756598 shutdown_connections 2024-04-25T01:23:08.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.100+0000 7f29e9328640 1 --2- 172.21.15.77:0/110756598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e40705c0 0x7f29e4166c30 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.100+0000 7f29e9328640 1 -- 172.21.15.77:0/110756598 >> 172.21.15.77:0/110756598 conn(0x7f29e40d1ca0 msgr2=0x7f29e40d20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:08.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.100+0000 7f29e9328640 1 -- 172.21.15.77:0/110756598 shutdown_connections 2024-04-25T01:23:08.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.100+0000 7f29e9328640 1 -- 172.21.15.77:0/110756598 wait complete. 2024-04-25T01:23:08.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.101+0000 7f29e9328640 1 Processor -- start 2024-04-25T01:23:08.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.101+0000 7f29e9328640 1 -- start start 2024-04-25T01:23:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.102+0000 7f29e9328640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e40705c0 0x7f29e406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.102+0000 7f29e9328640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e40d28a0 0x7f29e4157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.102+0000 7f29e9328640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e415f630 0x7f29e4157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.102+0000 7f29e9328640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f29e40b13f0 con 0x7f29e40705c0 2024-04-25T01:23:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.102+0000 7f29e9328640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f29e40cc6d0 con 0x7f29e415f630 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.102+0000 7f29e9328640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f29e406cc40 con 0x7f29e40d28a0 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.102+0000 7f29e2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e40705c0 0x7f29e406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.102+0000 7f29e2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e40705c0 0x7f29e406d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51186/0 (socket says 172.21.15.77:51186) 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.102+0000 7f29e2ffd640 1 -- 172.21.15.77:0/2397811250 learned_addr learned my addr 172.21.15.77:0/2397811250 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.102+0000 7f29e27fc640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e40d28a0 0x7f29e4157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29e37fe640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e415f630 0x7f29e4157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29e2ffd640 1 -- 172.21.15.77:0/2397811250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e40d28a0 msgr2=0x7f29e4157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29e2ffd640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e40d28a0 0x7f29e4157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29e2ffd640 1 -- 172.21.15.77:0/2397811250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e415f630 msgr2=0x7f29e4157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29e2ffd640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e415f630 0x7f29e4157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29e2ffd640 1 -- 172.21.15.77:0/2397811250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e4155ad0 con 0x7f29e40705c0 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29e2ffd640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e40705c0 0x7f29e406d3b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f29d400ba50 tx=0x7f29d400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29cb7fe640 1 -- 172.21.15.77:0/2397811250 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4019070 con 0x7f29e40705c0 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29cb7fe640 1 -- 172.21.15.77:0/2397811250 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29d400cc20 con 0x7f29e40705c0 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29cb7fe640 1 -- 172.21.15.77:0/2397811250 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4014680 con 0x7f29e40705c0 2024-04-25T01:23:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29e9328640 1 -- 172.21.15.77:0/2397811250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29e4155db0 con 0x7f29e40705c0 2024-04-25T01:23:08.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.103+0000 7f29e9328640 1 -- 172.21.15.77:0/2397811250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29e406ce90 con 0x7f29e40705c0 2024-04-25T01:23:08.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.104+0000 7f29e9328640 1 -- 172.21.15.77:0/2397811250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29e4068fb0 con 0x7f29e40705c0 2024-04-25T01:23:08.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.104+0000 7f29cb7fe640 1 -- 172.21.15.77:0/2397811250 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f29d401c070 con 0x7f29e40705c0 2024-04-25T01:23:08.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.105+0000 7f29cb7fe640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29d006be30 0x7f29d006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:08.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.105+0000 7f29cb7fe640 1 -- 172.21.15.77:0/2397811250 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f29d408ad80 con 0x7f29e40705c0 2024-04-25T01:23:08.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.105+0000 7f29e27fc640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29d006be30 0x7f29d006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:08.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.105+0000 7f29e27fc640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29d006be30 0x7f29d006e2f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f29e406df70 tx=0x7f29d8007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.109+0000 7f29cb7fe640 1 -- 172.21.15.77:0/2397811250 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f29d4058f00 con 0x7f29e40705c0 2024-04-25T01:23:08.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.319+0000 7f29e9328640 1 -- 172.21.15.77:0/2397811250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29e406d060 con 0x7f29e40705c0 2024-04-25T01:23:08.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.319+0000 7f29cb7fe640 1 -- 172.21.15.77:0/2397811250 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5181 (secure 0 0 0) 0x7f29d405d200 con 0x7f29e40705c0 2024-04-25T01:23:08.300 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:08.300 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2823778882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2823778882},{"type":"v1","addr":"172.21.15.77:6837","nonce":2823778882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.066906+0000","modified":"2024-04-25T01:23:07.248215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5355":{"gid":5355,"name":"c","rank":0,"incarnation":18,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.77:6835/3139470828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3139470828},{"type":"v1","addr":"172.21.15.77:6835","nonce":3139470828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T01:23:08.300 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2024-04-25T01:23:08.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.321+0000 7f29e9328640 1 -- 172.21.15.77:0/2397811250 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29d006be30 msgr2=0x7f29d006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:08.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.321+0000 7f29e9328640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29d006be30 0x7f29d006e2f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f29e406df70 tx=0x7f29d8007b50 comp rx=0 tx=0).stop 2024-04-25T01:23:08.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.321+0000 7f29e9328640 1 -- 172.21.15.77:0/2397811250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e40705c0 msgr2=0x7f29e406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:08.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.321+0000 7f29e9328640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e40705c0 0x7f29e406d3b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f29d400ba50 tx=0x7f29d400bf10 comp rx=0 tx=0).stop 2024-04-25T01:23:08.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.322+0000 7f29e9328640 1 -- 172.21.15.77:0/2397811250 shutdown_connections 2024-04-25T01:23:08.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.322+0000 7f29e9328640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f29d006be30 0x7f29d006e2f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.322+0000 7f29e9328640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e415f630 0x7f29e4157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.322+0000 7f29e9328640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e40d28a0 0x7f29e4157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.322+0000 7f29e9328640 1 --2- 172.21.15.77:0/2397811250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29e40705c0 0x7f29e406d3b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.322+0000 7f29e9328640 1 -- 172.21.15.77:0/2397811250 >> 172.21.15.77:0/2397811250 conn(0x7f29e40d1ca0 msgr2=0x7f29e40ae9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:08.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.322+0000 7f29e9328640 1 -- 172.21.15.77:0/2397811250 shutdown_connections 2024-04-25T01:23:08.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:08.322+0000 7f29e9328640 1 -- 172.21.15.77:0/2397811250 wait complete. 2024-04-25T01:23:08.715 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:22:57.066906+0000', 'modified': '2024-04-25T01:23:07.248215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5355}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5355': {'gid': 5355, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.77:6835/3139470828', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 3139470828}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 3139470828}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T01:23:08.716 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5355:up:rejoin 2024-04-25T01:23:09.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:09.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.845+0000 7f7c018dd640 1 Processor -- start 2024-04-25T01:23:09.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.845+0000 7f7c018dd640 1 -- start start 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7c018dd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7bfc162260 0x7f7bfc0b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7c018dd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bfc162640 0x7f7bfc162a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7c018dd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bfc0d49d0 con 0x7f7bfc161400 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7c018dd640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f7bfc0d4cb0 con 0x7f7bfc162260 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7c018dd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7bfc0cee60 con 0x7f7bfc162640 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7bfaffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7bfc162260 0x7f7bfc0b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7bfaffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7bfc162260 0x7f7bfc0b0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51188/0 (socket says 172.21.15.77:51188) 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7bfaffd640 1 -- 172.21.15.77:0/2870944575 learned_addr learned my addr 172.21.15.77:0/2870944575 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7bfa7fc640 1 --2- 172.21.15.77:0/2870944575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bfc162640 0x7f7bfc162a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7bfaffd640 1 -- 172.21.15.77:0/2870944575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bfc162640 msgr2=0x7f7bfc162a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7bfaffd640 1 --2- 172.21.15.77:0/2870944575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bfc162640 0x7f7bfc162a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7bfaffd640 1 -- 172.21.15.77:0/2870944575 >> v1:172.21.15.121:6789/0 conn(0x7f7bfc161400 legacy=0x7f7bfc0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.846+0000 7f7bfaffd640 1 -- 172.21.15.77:0/2870944575 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bfc0cf140 con 0x7f7bfc162260 2024-04-25T01:23:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.847+0000 7f7bfaffd640 1 --2- 172.21.15.77:0/2870944575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bfc162260 0x7f7bfc0b0bd0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009870 tx=0x7f7bec02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad6a46d382ffdbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.847+0000 7f7bf9ffb640 1 -- 172.21.15.77:0/2870944575 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec037070 con 0x7f7bfc162260 2024-04-25T01:23:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.847+0000 7f7bf9ffb640 1 -- 172.21.15.77:0/2870944575 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bec032440 con 0x7f7bfc162260 2024-04-25T01:23:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.847+0000 7f7bf9ffb640 1 -- 172.21.15.77:0/2870944575 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec032810 con 0x7f7bfc162260 2024-04-25T01:23:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.848+0000 7f7c018dd640 1 -- 172.21.15.77:0/2870944575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bfc162260 msgr2=0x7f7bfc0b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.848+0000 7f7c018dd640 1 --2- 172.21.15.77:0/2870944575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bfc162260 0x7f7bfc0b0bd0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009870 tx=0x7f7bec02a210 comp rx=0 tx=0).stop 2024-04-25T01:23:09.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.848+0000 7f7c018dd640 1 -- 172.21.15.77:0/2870944575 shutdown_connections 2024-04-25T01:23:09.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.848+0000 7f7c018dd640 1 --2- 172.21.15.77:0/2870944575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bfc162640 0x7f7bfc162a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.848+0000 7f7c018dd640 1 --2- 172.21.15.77:0/2870944575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bfc162260 0x7f7bfc0b0bd0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.848+0000 7f7c018dd640 1 -- 172.21.15.77:0/2870944575 >> 172.21.15.77:0/2870944575 conn(0x7f7bfc09d200 msgr2=0x7f7bfc0d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:09.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.848+0000 7f7c018dd640 1 -- 172.21.15.77:0/2870944575 shutdown_connections 2024-04-25T01:23:09.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.848+0000 7f7c018dd640 1 -- 172.21.15.77:0/2870944575 wait complete. 2024-04-25T01:23:09.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.849+0000 7f7c018dd640 1 Processor -- start 2024-04-25T01:23:09.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.849+0000 7f7c018dd640 1 -- start start 2024-04-25T01:23:09.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7c018dd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bfc161400 0x7f7bfc1f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:09.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7c018dd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bfc162260 0x7f7bfc0ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:09.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7c018dd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bfc162640 0x7f7bfc15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:09.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7c018dd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7bfc0cec20 con 0x7f7bfc161400 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7c018dd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7bfc0ceaa0 con 0x7f7bfc162640 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7c018dd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7bfc0b1110 con 0x7f7bfc162260 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bfc161400 0x7f7bfc1f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bfc161400 0x7f7bfc1f3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51196/0 (socket says 172.21.15.77:51196) 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfaffd640 1 -- 172.21.15.77:0/2856457711 learned_addr learned my addr 172.21.15.77:0/2856457711 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfb7fe640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bfc162640 0x7f7bfc15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfa7fc640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bfc162260 0x7f7bfc0ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfaffd640 1 -- 172.21.15.77:0/2856457711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bfc162260 msgr2=0x7f7bfc0ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfaffd640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bfc162260 0x7f7bfc0ca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfaffd640 1 -- 172.21.15.77:0/2856457711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bfc162640 msgr2=0x7f7bfc15b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfaffd640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bfc162640 0x7f7bfc15b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfaffd640 1 -- 172.21.15.77:0/2856457711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bfc0c84e0 con 0x7f7bfc161400 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfa7fc640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bfc162260 0x7f7bfc0ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.850+0000 7f7bfb7fe640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bfc162640 0x7f7bfc15b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.851+0000 7f7bfaffd640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bfc161400 0x7f7bfc1f3f10 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009870 tx=0x7f7bec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:09.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.851+0000 7f7be3fff640 1 -- 172.21.15.77:0/2856457711 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec041020 con 0x7f7bfc161400 2024-04-25T01:23:09.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.851+0000 7f7be3fff640 1 -- 172.21.15.77:0/2856457711 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bec0306c0 con 0x7f7bfc161400 2024-04-25T01:23:09.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.851+0000 7f7be3fff640 1 -- 172.21.15.77:0/2856457711 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec037040 con 0x7f7bfc161400 2024-04-25T01:23:09.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.851+0000 7f7c018dd640 1 -- 172.21.15.77:0/2856457711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bfc0c87c0 con 0x7f7bfc161400 2024-04-25T01:23:09.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.851+0000 7f7c018dd640 1 -- 172.21.15.77:0/2856457711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bfc0c8d80 con 0x7f7bfc161400 2024-04-25T01:23:09.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.852+0000 7f7c018dd640 1 -- 172.21.15.77:0/2856457711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bfc068fb0 con 0x7f7bfc161400 2024-04-25T01:23:09.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.852+0000 7f7be3fff640 1 -- 172.21.15.77:0/2856457711 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7bec044020 con 0x7f7bfc161400 2024-04-25T01:23:09.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.853+0000 7f7be3fff640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7be806be30 0x7f7be806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:09.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.853+0000 7f7be3fff640 1 -- 172.21.15.77:0/2856457711 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f7bec0ab8c0 con 0x7f7bfc161400 2024-04-25T01:23:09.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.853+0000 7f7bfa7fc640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7be806be30 0x7f7be806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:09.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.853+0000 7f7bfa7fc640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7be806be30 0x7f7be806e2f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc1560f0 tx=0x7f7bf0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:09.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:09.857+0000 7f7be3fff640 1 -- 172.21.15.77:0/2856457711 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7bec04b030 con 0x7f7bfc161400 2024-04-25T01:23:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.067+0000 7f7c018dd640 1 -- 172.21.15.77:0/2856457711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bfc0b3c10 con 0x7f7bfc161400 2024-04-25T01:23:10.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.068+0000 7f7be3fff640 1 -- 172.21.15.77:0/2856457711 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5188 (secure 0 0 0) 0x7f7bec07a090 con 0x7f7bfc161400 2024-04-25T01:23:10.048 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:10.048 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2823778882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2823778882},{"type":"v1","addr":"172.21.15.77:6837","nonce":2823778882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.066906+0000","modified":"2024-04-25T01:23:09.251757+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5355":{"gid":5355,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6835/3139470828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3139470828},{"type":"v1","addr":"172.21.15.77:6835","nonce":3139470828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5355,"qdb_cluster":[5355]},"id":2}]} 2024-04-25T01:23:10.049 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-04-25T01:23:10.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 -- 172.21.15.77:0/2856457711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7be806be30 msgr2=0x7f7be806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7be806be30 0x7f7be806e2f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc1560f0 tx=0x7f7bf0008040 comp rx=0 tx=0).stop 2024-04-25T01:23:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 -- 172.21.15.77:0/2856457711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bfc161400 msgr2=0x7f7bfc1f3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bfc161400 0x7f7bfc1f3f10 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009870 tx=0x7f7bec0046c0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 -- 172.21.15.77:0/2856457711 shutdown_connections 2024-04-25T01:23:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7be806be30 0x7f7be806e2f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bfc162640 0x7f7bfc15b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bfc162260 0x7f7bfc0ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 --2- 172.21.15.77:0/2856457711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bfc161400 0x7f7bfc1f3f10 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 -- 172.21.15.77:0/2856457711 >> 172.21.15.77:0/2856457711 conn(0x7f7bfc09d200 msgr2=0x7f7bfc0d2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 -- 172.21.15.77:0/2856457711 shutdown_connections 2024-04-25T01:23:10.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.070+0000 7f7c018dd640 1 -- 172.21.15.77:0/2856457711 wait complete. 2024-04-25T01:23:10.464 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:22:57.066906+0000', 'modified': '2024-04-25T01:23:09.251757+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5355}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5355': {'gid': 5355, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.77:6835/3139470828', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 3139470828}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 3139470828}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5355, 'qdb_cluster': [5355]} 2024-04-25T01:23:10.465 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:23:10.465 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T01:23:10.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.583+0000 7f08811d5640 1 Processor -- start 2024-04-25T01:23:10.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.583+0000 7f08811d5640 1 -- start start 2024-04-25T01:23:10.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f08811d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f087c154ac0 0x7f087c15b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f08811d5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f087c0cf2a0 0x7f087c0711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f08811d5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f087c154ea0 con 0x7f087c071730 2024-04-25T01:23:10.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f08811d5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f087c06d540 con 0x7f087c0cf2a0 2024-04-25T01:23:10.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f08811d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f087c06d790 con 0x7f087c154ac0 2024-04-25T01:23:10.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f0873fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f087c0cf2a0 0x7f087c0711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f0873fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f087c0cf2a0 0x7f087c0711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51210/0 (socket says 172.21.15.77:51210) 2024-04-25T01:23:10.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f0873fff640 1 -- 172.21.15.77:0/1493582847 learned_addr learned my addr 172.21.15.77:0/1493582847 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:10.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f087ad9d640 1 --2- 172.21.15.77:0/1493582847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f087c154ac0 0x7f087c15b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f0873fff640 1 -- 172.21.15.77:0/1493582847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f087c154ac0 msgr2=0x7f087c15b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f0873fff640 1 --2- 172.21.15.77:0/1493582847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f087c154ac0 0x7f087c15b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f0873fff640 1 -- 172.21.15.77:0/1493582847 >> v1:172.21.15.121:6789/0 conn(0x7f087c071730 legacy=0x7f087c06e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.584+0000 7f0873fff640 1 -- 172.21.15.77:0/1493582847 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087c06da70 con 0x7f087c0cf2a0 2024-04-25T01:23:10.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.585+0000 7f0873fff640 1 --2- 172.21.15.77:0/1493582847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf2a0 0x7f087c0711f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f086400aa10 tx=0x7f086402b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4deeaa6da834a2b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.585+0000 7f087a59c640 1 -- 172.21.15.77:0/1493582847 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086400b040 con 0x7f087c0cf2a0 2024-04-25T01:23:10.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.585+0000 7f087a59c640 1 -- 172.21.15.77:0/1493582847 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0864030710 con 0x7f087c0cf2a0 2024-04-25T01:23:10.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.585+0000 7f087a59c640 1 -- 172.21.15.77:0/1493582847 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864030ae0 con 0x7f087c0cf2a0 2024-04-25T01:23:10.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.585+0000 7f08811d5640 1 -- 172.21.15.77:0/1493582847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf2a0 msgr2=0x7f087c0711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.585+0000 7f08811d5640 1 --2- 172.21.15.77:0/1493582847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf2a0 0x7f087c0711f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f086400aa10 tx=0x7f086402b5d0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.586+0000 7f08811d5640 1 -- 172.21.15.77:0/1493582847 shutdown_connections 2024-04-25T01:23:10.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.586+0000 7f08811d5640 1 --2- 172.21.15.77:0/1493582847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf2a0 0x7f087c0711f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.586+0000 7f08811d5640 1 --2- 172.21.15.77:0/1493582847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f087c154ac0 0x7f087c15b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.586+0000 7f08811d5640 1 -- 172.21.15.77:0/1493582847 >> 172.21.15.77:0/1493582847 conn(0x7f087c0b1820 msgr2=0x7f087c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.586+0000 7f08811d5640 1 -- 172.21.15.77:0/1493582847 shutdown_connections 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.586+0000 7f08811d5640 1 -- 172.21.15.77:0/1493582847 wait complete. 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.587+0000 7f08811d5640 1 Processor -- start 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.587+0000 7f08811d5640 1 -- start start 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f08811d5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f087c071730 0x7f087c1f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f08811d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf2a0 0x7f087c14caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f08811d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f087c154ac0 0x7f087c145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f08811d5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f087c0cf720 con 0x7f087c0cf2a0 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f08811d5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f087c06e030 con 0x7f087c071730 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f08811d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f087c06ec60 con 0x7f087c154ac0 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f0873fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf2a0 0x7f087c14caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f0873fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf2a0 0x7f087c14caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51224/0 (socket says 172.21.15.77:51224) 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f0873fff640 1 -- 172.21.15.77:0/3241714247 learned_addr learned my addr 172.21.15.77:0/3241714247 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:10.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f087ad9d640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f087c071730 0x7f087c1f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f087b59e640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f087c154ac0 0x7f087c145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f0873fff640 1 -- 172.21.15.77:0/3241714247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f087c154ac0 msgr2=0x7f087c145620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f0873fff640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f087c154ac0 0x7f087c145620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f0873fff640 1 -- 172.21.15.77:0/3241714247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f087c071730 msgr2=0x7f087c1f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f0873fff640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f087c071730 0x7f087c1f4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f0873fff640 1 -- 172.21.15.77:0/3241714247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087c145b60 con 0x7f087c0cf2a0 2024-04-25T01:23:10.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f087ad9d640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f087c071730 0x7f087c1f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:10.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f0873fff640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf2a0 0x7f087c14caf0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f086400a1c0 tx=0x7f08640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f08737fe640 1 -- 172.21.15.77:0/3241714247 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086403f020 con 0x7f087c0cf2a0 2024-04-25T01:23:10.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f08737fe640 1 -- 172.21.15.77:0/3241714247 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0864030e30 con 0x7f087c0cf2a0 2024-04-25T01:23:10.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f08737fe640 1 -- 172.21.15.77:0/3241714247 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086403b430 con 0x7f087c0cf2a0 2024-04-25T01:23:10.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f08811d5640 1 -- 172.21.15.77:0/3241714247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f087c0ad8e0 con 0x7f087c0cf2a0 2024-04-25T01:23:10.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.588+0000 7f08811d5640 1 -- 172.21.15.77:0/3241714247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f087c1f4d40 con 0x7f087c0cf2a0 2024-04-25T01:23:10.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.590+0000 7f08737fe640 1 -- 172.21.15.77:0/3241714247 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0864042020 con 0x7f087c0cf2a0 2024-04-25T01:23:10.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.590+0000 7f08737fe640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f085c06be30 0x7f085c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.590+0000 7f08737fe640 1 -- 172.21.15.77:0/3241714247 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f08640ab910 con 0x7f087c0cf2a0 2024-04-25T01:23:10.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.590+0000 7f087ad9d640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f085c06be30 0x7f085c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.591+0000 7f087ad9d640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f085c06be30 0x7f085c06e2f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f086c006fd0 tx=0x7f086c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.591+0000 7f0883fc4740 1 Processor -- start 2024-04-25T01:23:10.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.591+0000 7f0883fc4740 1 -- start start 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba164f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba46000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46540 0x5653eba1b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0883fc4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5653eb97ef00 con 0x5653eba46c80 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0883fc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5653eb927750 con 0x5653eba63b60 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0883fc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5653eb97aec0 con 0x5653eba46540 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0873fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba46000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0873fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba46000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51232/0 (socket says 172.21.15.77:51232) 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0873fff640 1 -- 172.21.15.77:0/3430348214 learned_addr learned my addr 172.21.15.77:0/3430348214 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f087b59e640 1 --2- 172.21.15.77:0/3430348214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46540 0x5653eba1b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f087ad9d640 1 --2- 172.21.15.77:0/3430348214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba164f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0873fff640 1 -- 172.21.15.77:0/3430348214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46540 msgr2=0x5653eba1b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0873fff640 1 --2- 172.21.15.77:0/3430348214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46540 0x5653eba1b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0873fff640 1 -- 172.21.15.77:0/3430348214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 msgr2=0x5653eba164f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0873fff640 1 --2- 172.21.15.77:0/3430348214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba164f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.592+0000 7f0873fff640 1 -- 172.21.15.77:0/3430348214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653eb6932d0 con 0x5653eba46c80 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.593+0000 7f0873fff640 1 --2- 172.21.15.77:0/3430348214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba46000 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f08640a9c00 tx=0x7f086400c0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6887fe0cce518894 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.593+0000 7f0870ff9640 1 -- 172.21.15.77:0/3430348214 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08640825c0 con 0x5653eba46c80 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.593+0000 7f0870ff9640 1 -- 172.21.15.77:0/3430348214 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0864077d90 con 0x5653eba46c80 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.593+0000 7f0870ff9640 1 -- 172.21.15.77:0/3430348214 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08640800c0 con 0x5653eba46c80 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.593+0000 7f0883fc4740 1 -- 172.21.15.77:0/3430348214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 msgr2=0x5653eba46000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.593+0000 7f0883fc4740 1 --2- 172.21.15.77:0/3430348214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba46000 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f08640a9c00 tx=0x7f086400c0d0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.593+0000 7f0883fc4740 1 -- 172.21.15.77:0/3430348214 shutdown_connections 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.593+0000 7f0883fc4740 1 --2- 172.21.15.77:0/3430348214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba46000 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.593+0000 7f0883fc4740 1 --2- 172.21.15.77:0/3430348214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46540 0x5653eba1b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.593+0000 7f0883fc4740 1 --2- 172.21.15.77:0/3430348214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba164f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.593+0000 7f0883fc4740 1 -- 172.21.15.77:0/3430348214 >> 172.21.15.77:0/3430348214 conn(0x5653eba18760 msgr2=0x5653eba61000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.594+0000 7f0883fc4740 1 -- 172.21.15.77:0/3430348214 shutdown_connections 2024-04-25T01:23:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.594+0000 7f0883fc4740 1 -- 172.21.15.77:0/3430348214 wait complete. 2024-04-25T01:23:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.594+0000 7f0883fc4740 1 -- start start 2024-04-25T01:23:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.594+0000 7f0883fc4740 1 Processor -- start 2024-04-25T01:23:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46c80 0x5653eba49250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba62f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba45490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f0883fc4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5653eb927750 con 0x5653eba45010 2024-04-25T01:23:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f0883fc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5653eb93acf0 con 0x5653eba63b60 2024-04-25T01:23:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f0883fc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5653eb97ef00 con 0x5653eba46c80 2024-04-25T01:23:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f0883fc4740 10 client.0 _ll_register_callbacks cb 0x5653eb93a6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:23:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f087b59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba45490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f087b59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba45490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51238/0 (socket says 172.21.15.77:51238) 2024-04-25T01:23:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f087b59e640 1 -- 172.21.15.77:0/4181139632 learned_addr learned my addr 172.21.15.77:0/4181139632 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f0873fff640 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba62f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f087ad9d640 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46c80 0x5653eba49250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f087b59e640 1 -- 172.21.15.77:0/4181139632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46c80 msgr2=0x5653eba49250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f087b59e640 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46c80 0x5653eba49250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f087b59e640 1 -- 172.21.15.77:0/4181139632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 msgr2=0x5653eba62f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f087b59e640 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba62f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f087b59e640 1 -- 172.21.15.77:0/4181139632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653eb75d6b0 con 0x5653eba45010 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f0873fff640 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba62f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f087b59e640 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba45490 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f087400cba0 tx=0x7f087400a3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24740eb86d1511e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.595+0000 7f0859ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0859ffb640 1 -- 172.21.15.77:0/4181139632 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087401b020 con 0x5653eba45010 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0859ffb640 1 -- 172.21.15.77:0/4181139632 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f0874003970 con 0x5653eba45010 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0883fc4740 1 -- 172.21.15.77:0/4181139632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0864077d90 con 0x5653eba45010 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0859ffb640 1 -- 172.21.15.77:0/4181139632 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0874022050 con 0x5653eba45010 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0883fc4740 10 client.5643 fetch_fsmap learned FSMap version 21 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0883fc4740 1 -- 172.21.15.77:0/4181139632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x5653eba47f40 con 0x5653eba45010 2024-04-25T01:23:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0859ffb640 1 -- 172.21.15.77:0/4181139632 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0864077d90 con 0x5653eba45010 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0859ffb640 1 -- 172.21.15.77:0/4181139632 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7f0874025020 con 0x5653eba45010 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0883fc4740 10 client.5643 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0883fc4740 10 client.5643 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0883fc4740 20 client.5643 populate_metadata read hostname 'smithi077' 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0883fc4740 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.77:6834/3139470828,v1:172.21.15.77:6835/3139470828] conn(0x5653eba61480 0x5653eba71770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0883fc4740 4 client.5643 mds_command: new command op to 5355 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0883fc4740 1 -- 172.21.15.77:0/4181139632 --> [v2:172.21.15.77:6834/3139470828,v1:172.21.15.77:6835/3139470828] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653eb98c440 con 0x5653eba61480 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f087ad9d640 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.77:6834/3139470828,v1:172.21.15.77:6835/3139470828] conn(0x5653eba61480 0x5653eba71770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f0859ffb640 10 client.5643 ms_handle_connect on v2:172.21.15.77:6834/3139470828 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.596+0000 7f087ad9d640 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.77:6834/3139470828,v1:172.21.15.77:6835/3139470828] conn(0x5653eba61480 0x5653eba71770 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d82bb56a5a7b5e83 server_cookie=743fc6343cdde82a in_seq=0 out_seq=0 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0859ffb640 1 -- 172.21.15.77:0/4181139632 <== mds.0 v2:172.21.15.77:6834/3139470828 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653eb98c440 con 0x5653eba61480 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0859ffb640 10 client.5643 handle_command_reply: tid=0 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 1 client.5643 shutdown 2024-04-25T01:23:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 1 -- 172.21.15.77:0/4181139632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 msgr2=0x5653eba45490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba45490 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f087400cba0 tx=0x7f087400a3e0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 1 -- 172.21.15.77:0/4181139632 shutdown_connections 2024-04-25T01:23:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.77:6834/3139470828,v1:172.21.15.77:6835/3139470828] conn(0x5653eba61480 0x5653eba71770 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba45490 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0859ffb640 0 client.5643 ms_handle_reset on v2:172.21.15.77:6834/3139470828 2024-04-25T01:23:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba62f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 1 --2- 172.21.15.77:0/4181139632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46c80 0x5653eba49250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 1 -- 172.21.15.77:0/4181139632 >> 172.21.15.77:0/4181139632 conn(0x5653eba46540 msgr2=0x5653eba5e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 1 -- 172.21.15.77:0/4181139632 shutdown_connections 2024-04-25T01:23:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 1 -- 172.21.15.77:0/4181139632 wait complete. 2024-04-25T01:23:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.597+0000 7f0883fc4740 20 client.5643 trim_cache size 0 max 16384 2024-04-25T01:23:10.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.615+0000 7f0883fc4740 1 Processor -- start 2024-04-25T01:23:10.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.615+0000 7f0883fc4740 1 -- start start 2024-04-25T01:23:10.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba1bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46c80 0x5653eba72580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba72ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f0883fc4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1ffe00 con 0x5653eba45010 2024-04-25T01:23:10.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f0883fc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1598e0 con 0x5653eba63b60 2024-04-25T01:23:10.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f0883fc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f087c159f30 con 0x5653eba46c80 2024-04-25T01:23:10.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f087ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba1bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f087ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba1bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51242/0 (socket says 172.21.15.77:51242) 2024-04-25T01:23:10.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f087ad9d640 1 -- 172.21.15.77:0/3104623190 learned_addr learned my addr 172.21.15.77:0/3104623190 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:10.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f087ad9d640 1 -- 172.21.15.77:0/3104623190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46c80 msgr2=0x5653eba72580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f087ad9d640 1 --2- 172.21.15.77:0/3104623190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46c80 0x5653eba72580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f087ad9d640 1 -- 172.21.15.77:0/3104623190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 msgr2=0x5653eba72ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f087ad9d640 1 --2- 172.21.15.77:0/3104623190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba72ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.616+0000 7f087ad9d640 1 -- 172.21.15.77:0/3104623190 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653eba70d90 con 0x5653eba45010 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f087ad9d640 1 --2- 172.21.15.77:0/3104623190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba1bf00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f086c009720 tx=0x7f086c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c98d551b95867c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f085bfff640 1 -- 172.21.15.77:0/3104623190 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c02d070 con 0x5653eba45010 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f085bfff640 1 -- 172.21.15.77:0/3104623190 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f086c0270f0 con 0x5653eba45010 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f085bfff640 1 -- 172.21.15.77:0/3104623190 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c00f040 con 0x5653eba45010 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f0883fc4740 1 -- 172.21.15.77:0/3104623190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 msgr2=0x5653eba1bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f0883fc4740 1 --2- 172.21.15.77:0/3104623190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba1bf00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f086c009720 tx=0x7f086c013000 comp rx=0 tx=0).stop 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f0883fc4740 1 -- 172.21.15.77:0/3104623190 shutdown_connections 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f0883fc4740 1 --2- 172.21.15.77:0/3104623190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba63b60 0x5653eba72ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f0883fc4740 1 --2- 172.21.15.77:0/3104623190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba46c80 0x5653eba72580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f0883fc4740 1 --2- 172.21.15.77:0/3104623190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba45010 0x5653eba1bf00 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f0883fc4740 1 -- 172.21.15.77:0/3104623190 >> 172.21.15.77:0/3104623190 conn(0x5653eba46540 msgr2=0x5653eba11690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f0883fc4740 1 -- 172.21.15.77:0/3104623190 shutdown_connections 2024-04-25T01:23:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.617+0000 7f0883fc4740 1 -- 172.21.15.77:0/3104623190 wait complete. 2024-04-25T01:23:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.618+0000 7f0883fc4740 1 -- start start 2024-04-25T01:23:10.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.618+0000 7f0883fc4740 1 Processor -- start 2024-04-25T01:23:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.618+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba45010 0x5653eba73800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.618+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba713c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.618+0000 7f0883fc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba63b60 0x5653eba71900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.618+0000 7f0883fc4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5653eba72360 con 0x5653eba46c80 2024-04-25T01:23:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.618+0000 7f0883fc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5653eba48270 con 0x5653eba45010 2024-04-25T01:23:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.618+0000 7f0883fc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5653eba71240 con 0x5653eba63b60 2024-04-25T01:23:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0883fc4740 10 client.0 _ll_register_callbacks cb 0x7f087400c7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:23:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0873fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0873fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba713c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51250/0 (socket says 172.21.15.77:51250) 2024-04-25T01:23:10.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0873fff640 1 -- 172.21.15.77:0/604766917 learned_addr learned my addr 172.21.15.77:0/604766917 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f087ad9d640 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba45010 0x5653eba73800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f087b59e640 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba63b60 0x5653eba71900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0873fff640 1 -- 172.21.15.77:0/604766917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba63b60 msgr2=0x5653eba71900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0873fff640 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba63b60 0x5653eba71900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0873fff640 1 -- 172.21.15.77:0/604766917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba45010 msgr2=0x5653eba73800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0873fff640 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba45010 0x5653eba73800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0873fff640 1 -- 172.21.15.77:0/604766917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653eba73d40 con 0x5653eba46c80 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0873fff640 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba713c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f08640a9cb0 tx=0x7f086400a790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d49a514e6ec7c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0859ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0859ffb640 1 -- 172.21.15.77:0/604766917 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864075d30 con 0x5653eba46c80 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0859ffb640 1 -- 172.21.15.77:0/604766917 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f0864082130 con 0x5653eba46c80 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0883fc4740 1 -- 172.21.15.77:0/604766917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f086c0270f0 con 0x5653eba46c80 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0859ffb640 1 -- 172.21.15.77:0/604766917 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f08640802c0 con 0x5653eba46c80 2024-04-25T01:23:10.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0883fc4740 10 client.5649 fetch_fsmap learned FSMap version 21 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0883fc4740 1 -- 172.21.15.77:0/604766917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x5653eba58000 con 0x5653eba46c80 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0859ffb640 1 -- 172.21.15.77:0/604766917 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f086c0270f0 con 0x5653eba46c80 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.619+0000 7f0859ffb640 1 -- 172.21.15.77:0/604766917 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7f08640a30d0 con 0x5653eba46c80 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 10 client.5649 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 10 client.5649 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 20 client.5649 populate_metadata read hostname 'smithi077' 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.77:6834/3139470828,v1:172.21.15.77:6835/3139470828] conn(0x5653ebadb390 0x5653eba83840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 4 client.5649 mds_command: new command op to 5355 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 -- 172.21.15.77:0/604766917 --> [v2:172.21.15.77:6834/3139470828,v1:172.21.15.77:6835/3139470828] -- command(tid 0: {"prefix": "status"}) -- 0x5653eb986790 con 0x5653ebadb390 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f087ad9d640 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.77:6834/3139470828,v1:172.21.15.77:6835/3139470828] conn(0x5653ebadb390 0x5653eba83840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0859ffb640 10 client.5649 ms_handle_connect on v2:172.21.15.77:6834/3139470828 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f087ad9d640 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.77:6834/3139470828,v1:172.21.15.77:6835/3139470828] conn(0x5653ebadb390 0x5653eba83840 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a6e855f3233cf77 server_cookie=4c639d6e8e5c6265 in_seq=0 out_seq=0 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0859ffb640 1 -- 172.21.15.77:0/604766917 <== mds.0 v2:172.21.15.77:6834/3139470828 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5653eb986790 con 0x5653ebadb390 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0859ffb640 10 client.5649 handle_command_reply: tid=0 2024-04-25T01:23:10.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 client.5649 shutdown 2024-04-25T01:23:10.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 -- 172.21.15.77:0/604766917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 msgr2=0x5653eba713c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba713c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f08640a9cb0 tx=0x7f086400a790 comp rx=0 tx=0).stop 2024-04-25T01:23:10.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 -- 172.21.15.77:0/604766917 shutdown_connections 2024-04-25T01:23:10.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.77:6834/3139470828,v1:172.21.15.77:6835/3139470828] conn(0x5653ebadb390 0x5653eba83840 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653eba46c80 0x5653eba713c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653eba63b60 0x5653eba71900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0859ffb640 0 client.5649 ms_handle_reset on v2:172.21.15.77:6834/3139470828 2024-04-25T01:23:10.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 --2- 172.21.15.77:0/604766917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653eba45010 0x5653eba73800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 -- 172.21.15.77:0/604766917 >> 172.21.15.77:0/604766917 conn(0x5653eba46540 msgr2=0x5653eba162f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 -- 172.21.15.77:0/604766917 shutdown_connections 2024-04-25T01:23:10.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 1 -- 172.21.15.77:0/604766917 wait complete. 2024-04-25T01:23:10.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.620+0000 7f0883fc4740 20 client.5649 trim_cache size 0 max 16384 2024-04-25T01:23:10.611 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:23:10.611 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "7c3c39ee-077e-476c-8994-eede4fab301d", 2024-04-25T01:23:10.611 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-04-25T01:23:10.612 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 5355, 2024-04-25T01:23:10.612 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-04-25T01:23:10.612 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-04-25T01:23:10.612 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-04-25T01:23:10.612 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 4.3759056779999996, 2024-04-25T01:23:10.612 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 21, 2024-04-25T01:23:10.612 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 39, 2024-04-25T01:23:10.612 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 39, 2024-04-25T01:23:10.612 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 19.788863082999999 2024-04-25T01:23:10.612 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:23:10.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 -- 172.21.15.77:0/3241714247 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f085c06be30 msgr2=0x7f085c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f085c06be30 0x7f085c06e2f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f086c006fd0 tx=0x7f086c008040 comp rx=0 tx=0).stop 2024-04-25T01:23:10.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 -- 172.21.15.77:0/3241714247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf2a0 msgr2=0x7f087c14caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf2a0 0x7f087c14caf0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f086400a1c0 tx=0x7f08640046c0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 -- 172.21.15.77:0/3241714247 shutdown_connections 2024-04-25T01:23:10.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f085c06be30 0x7f085c06e2f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf2a0 0x7f087c14caf0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f087c154ac0 0x7f087c145620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 --2- 172.21.15.77:0/3241714247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f087c071730 0x7f087c1f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 -- 172.21.15.77:0/3241714247 >> 172.21.15.77:0/3241714247 conn(0x7f087c0b1820 msgr2=0x7f087c0b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 -- 172.21.15.77:0/3241714247 shutdown_connections 2024-04-25T01:23:10.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.621+0000 7f08811d5640 1 -- 172.21.15.77:0/3241714247 wait complete. 2024-04-25T01:23:10.615 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000000.00000000 parent 2024-04-25T01:23:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62d516f40 1 Processor -- start 2024-04-25T01:23:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62d516f40 1 -- start start 2024-04-25T01:23:10.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62d516f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ef279d59c0 0x55ef27ab0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62d516f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef27aac430 0x55ef27aa8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62d516f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ef27ab22f0 0x55ef27a05d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62d516f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef27a214d0 con 0x55ef27aac430 2024-04-25T01:23:10.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62d516f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ef2797e640 con 0x55ef27ab22f0 2024-04-25T01:23:10.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62d516f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ef27a21040 con 0x55ef279d59c0 2024-04-25T01:23:10.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62a818640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef27aac430 0x55ef27aa8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62a818640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef27aac430 0x55ef27aa8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51276/0 (socket says 172.21.15.77:51276) 2024-04-25T01:23:10.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62a818640 1 -- 172.21.15.77:0/1335288428 learned_addr learned my addr 172.21.15.77:0/1335288428 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:10.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62b81a640 1 --2- 172.21.15.77:0/1335288428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ef27ab22f0 0x55ef27a05d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.677+0000 7fe62b019640 1 --2- 172.21.15.77:0/1335288428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ef279d59c0 0x55ef27ab0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62a818640 1 -- 172.21.15.77:0/1335288428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ef279d59c0 msgr2=0x55ef27ab0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62a818640 1 --2- 172.21.15.77:0/1335288428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ef279d59c0 0x55ef27ab0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62a818640 1 -- 172.21.15.77:0/1335288428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ef27ab22f0 msgr2=0x55ef27a05d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62a818640 1 --2- 172.21.15.77:0/1335288428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ef27ab22f0 0x55ef27a05d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62a818640 1 -- 172.21.15.77:0/1335288428 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef27ab1dc0 con 0x55ef27aac430 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62b81a640 1 --2- 172.21.15.77:0/1335288428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ef27ab22f0 0x55ef27a05d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62a818640 1 --2- 172.21.15.77:0/1335288428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef27aac430 0x55ef27aa8120 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe620045a20 tx=0x7fe620051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baee13eddd0a6d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe629816640 1 -- 172.21.15.77:0/1335288428 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620053ad0 con 0x55ef27aac430 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe629816640 1 -- 172.21.15.77:0/1335288428 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe620063030 con 0x55ef27aac430 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe629816640 1 -- 172.21.15.77:0/1335288428 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620068420 con 0x55ef27aac430 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62d516f40 1 -- 172.21.15.77:0/1335288428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef27aac430 msgr2=0x55ef27aa8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62d516f40 1 --2- 172.21.15.77:0/1335288428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef27aac430 0x55ef27aa8120 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe620045a20 tx=0x7fe620051000 comp rx=0 tx=0).stop 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62d516f40 1 -- 172.21.15.77:0/1335288428 shutdown_connections 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62d516f40 1 --2- 172.21.15.77:0/1335288428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef27aac430 0x55ef27aa8120 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62d516f40 1 --2- 172.21.15.77:0/1335288428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ef27ab22f0 0x55ef27a05d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62d516f40 1 --2- 172.21.15.77:0/1335288428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ef279d59c0 0x55ef27ab0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62d516f40 1 -- 172.21.15.77:0/1335288428 >> 172.21.15.77:0/1335288428 conn(0x55ef27a01740 msgr2=0x55ef27aab6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62d516f40 1 -- 172.21.15.77:0/1335288428 shutdown_connections 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.678+0000 7fe62d516f40 1 -- 172.21.15.77:0/1335288428 wait complete. 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.679+0000 7fe62d516f40 1 Processor -- start 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.679+0000 7fe62d516f40 1 -- start start 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62d516f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef279d59c0 0x55ef27b44e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62d516f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ef27aac430 0x55ef27aaa520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62d516f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ef27ab22f0 0x55ef27aaaa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62d516f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef279d95d0 con 0x55ef279d59c0 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62d516f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ef279d9380 con 0x55ef27aac430 2024-04-25T01:23:10.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62d516f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ef279d98b0 con 0x55ef27ab22f0 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62b019640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef279d59c0 0x55ef27b44e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62b019640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef279d59c0 0x55ef27b44e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51286/0 (socket says 172.21.15.77:51286) 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62b019640 1 -- 172.21.15.77:0/1302074867 learned_addr learned my addr 172.21.15.77:0/1302074867 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62b81a640 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ef27ab22f0 0x55ef27aaaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62a818640 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ef27aac430 0x55ef27aaa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62b019640 1 -- 172.21.15.77:0/1302074867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ef27ab22f0 msgr2=0x55ef27aaaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62b019640 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ef27ab22f0 0x55ef27aaaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62b019640 1 -- 172.21.15.77:0/1302074867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ef27aac430 msgr2=0x55ef27aaa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62b019640 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ef27aac430 0x55ef27aaa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62b019640 1 -- 172.21.15.77:0/1302074867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef27b453c0 con 0x55ef279d59c0 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62b019640 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef279d59c0 0x55ef27b44e80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe61c007ea0 tx=0x7fe61c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe628814640 1 -- 172.21.15.77:0/1302074867 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c018070 con 0x55ef279d59c0 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe628814640 1 -- 172.21.15.77:0/1302074867 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe61c011700 con 0x55ef279d59c0 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62d516f40 1 -- 172.21.15.77:0/1302074867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ef27a1b810 con 0x55ef279d59c0 2024-04-25T01:23:10.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe628814640 1 -- 172.21.15.77:0/1302074867 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c011ab0 con 0x55ef279d59c0 2024-04-25T01:23:10.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.680+0000 7fe62d516f40 1 -- 172.21.15.77:0/1302074867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef27aade60 con 0x55ef279d59c0 2024-04-25T01:23:10.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.682+0000 7fe628814640 1 -- 172.21.15.77:0/1302074867 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe61c01b070 con 0x55ef279d59c0 2024-04-25T01:23:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.682+0000 7fe628814640 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe61806a810 0x7fe61806ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.682+0000 7fe628814640 1 -- 172.21.15.77:0/1302074867 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7fe61c089090 con 0x55ef279d59c0 2024-04-25T01:23:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.682+0000 7fe62a818640 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe61806a810 0x7fe61806ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.682+0000 7fe62d516f40 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x55ef27aa3040 0x55ef27aa9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.682+0000 7fe62d516f40 1 -- 172.21.15.77:0/1302074867 --> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] -- osd_op(unknown.0.0:1 5.10 5:0cf6ad0f:::10000000000.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) -- 0x55ef27aa3420 con 0x55ef27aa3040 2024-04-25T01:23:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.682+0000 7fe62a818640 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe61806a810 0x7fe61806ccd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe620000c00 tx=0x7fe620053000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.682+0000 7fe62b81a640 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x55ef27aa3040 0x55ef27aa9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.683+0000 7fe62b81a640 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x55ef27aa3040 0x55ef27aa9260 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.684+0000 7fe62b81a640 1 -- 172.21.15.77:0/1302074867 <== osd.7 v2:172.21.15.121:6800/251765690 1 ==== osd_op_reply(1 10000000000.00000000 [rmxattr] v39'11 uv11 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe624010250 con 0x55ef27aa3040 2024-04-25T01:23:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.684+0000 7fe62d516f40 1 -- 172.21.15.77:0/1302074867 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x55ef27aa3040 msgr2=0x55ef27aa9260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.684+0000 7fe62d516f40 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x55ef27aa3040 0x55ef27aa9260 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.684+0000 7fe62d516f40 1 -- 172.21.15.77:0/1302074867 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe61806a810 msgr2=0x7fe61806ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.684+0000 7fe62d516f40 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe61806a810 0x7fe61806ccd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe620000c00 tx=0x7fe620053000 comp rx=0 tx=0).stop 2024-04-25T01:23:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.684+0000 7fe62d516f40 1 -- 172.21.15.77:0/1302074867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef279d59c0 msgr2=0x55ef27b44e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.684+0000 7fe62d516f40 1 --2- 172.21.15.77:0/1302074867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef279d59c0 0x55ef27b44e80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe61c007ea0 tx=0x7fe61c007f90 comp rx=0 tx=0).stop 2024-04-25T01:23:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.684+0000 7fe62b81a640 1 -- 172.21.15.77:0/1302074867 reap_dead start 2024-04-25T01:23:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.685+0000 7fe62d516f40 1 -- 172.21.15.77:0/1302074867 shutdown_connections 2024-04-25T01:23:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.685+0000 7fe62d516f40 1 -- 172.21.15.77:0/1302074867 >> 172.21.15.77:0/1302074867 conn(0x55ef27a01740 msgr2=0x55ef27a06200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.685+0000 7fe62d516f40 1 -- 172.21.15.77:0/1302074867 shutdown_connections 2024-04-25T01:23:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.685+0000 7fe62d516f40 1 -- 172.21.15.77:0/1302074867 wait complete. 2024-04-25T01:23:10.671 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:23:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.825+0000 7f760029e640 1 Processor -- start 2024-04-25T01:23:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.825+0000 7f760029e640 1 -- start start 2024-04-25T01:23:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.826+0000 7f760029e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80cfaf0 0x7f75f80d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.826+0000 7f760029e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f80d1e90 con 0x7f75f815e4c0 2024-04-25T01:23:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.826+0000 7f760029e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f80b54a0 con 0x7f75f80d4a10 2024-04-25T01:23:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.826+0000 7f760029e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f75f80d2130 con 0x7f75f80cfaf0 2024-04-25T01:23:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.826+0000 7f75fe03a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f75f815e4c0 0x7f75f80d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44870/0 (socket says 172.21.15.77:44870) 2024-04-25T01:23:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.826+0000 7f75fe03a640 1 -- 172.21.15.77:0/874269598 learned_addr learned my addr 172.21.15.77:0/874269598 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fe83b640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80cfaf0 0x7f75f80d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2191420436 0 0) 0x7f75f80d1e90 con 0x7f75f815e4c0 2024-04-25T01:23:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75e4003520 con 0x7f75f815e4c0 2024-04-25T01:23:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3190212510 0 0) 0x7f75e4003520 con 0x7f75f815e4c0 2024-04-25T01:23:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80cfaf0 msgr2=0x7f75f80d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80cfaf0 0x7f75f80d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 >> v1:172.21.15.121:6789/0 conn(0x7f75f80d4a10 legacy=0x7f75f80b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f80b5780 con 0x7f75f815e4c0 2024-04-25T01:23:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f75f40019f0 con 0x7f75f815e4c0 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 >> v1:172.21.15.77:6789/0 conn(0x7f75f815e4c0 legacy=0x7f75f80d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 0x7f75e4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80cfaf0 0x7f75e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e4004d40 0x7f75e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f75f80d2130 con 0x7f75e40039e0 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f75e400baf0 con 0x7f75e4004d40 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f75e400bcf0 con 0x7f75f80cfaf0 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fe03a640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 0x7f75e4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fe83b640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e4004d40 0x7f75e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd839640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80cfaf0 0x7f75e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fe03a640 1 -- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80cfaf0 msgr2=0x7f75e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fe03a640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80cfaf0 0x7f75e4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fe03a640 1 -- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e4004d40 msgr2=0x7f75e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fe03a640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e4004d40 0x7f75e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fe03a640 1 -- 172.21.15.77:0/874269598 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e4003520 con 0x7f75e40039e0 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fd839640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80cfaf0 0x7f75e4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.827+0000 7f75fe83b640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e4004d40 0x7f75e40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.828+0000 7f75fe03a640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 0x7f75e4003e00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f75f400a850 tx=0x7f75e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8681cdf14e13fce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.828+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f400a4a0 con 0x7f75e40039e0 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.828+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f402f070 con 0x7f75e40039e0 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.828+0000 7f75fd038640 1 -- 172.21.15.77:0/874269598 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f402bc20 con 0x7f75e40039e0 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.829+0000 7f760029e640 1 -- 172.21.15.77:0/874269598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 msgr2=0x7f75e4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.829+0000 7f760029e640 1 --2- 172.21.15.77:0/874269598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 0x7f75e4003e00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f75f400a850 tx=0x7f75e4005830 comp rx=0 tx=0).stop 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.829+0000 7f75fe83b640 1 -- 172.21.15.77:0/874269598 reap_dead start 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.829+0000 7f760029e640 1 -- 172.21.15.77:0/874269598 shutdown_connections 2024-04-25T01:23:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.829+0000 7f760029e640 1 -- 172.21.15.77:0/874269598 >> 172.21.15.77:0/874269598 conn(0x7f75f80d3ef0 msgr2=0x7f75f80d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.829+0000 7f760029e640 1 -- 172.21.15.77:0/874269598 shutdown_connections 2024-04-25T01:23:10.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.829+0000 7f760029e640 1 -- 172.21.15.77:0/874269598 wait complete. 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.830+0000 7f760029e640 1 Processor -- start 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.830+0000 7f760029e640 1 -- start start 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f760029e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 0x7f75f8147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f760029e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80c9bf0 0x7f75f80ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f760029e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f8148530 0x7f75f81489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f760029e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f75f80b30a0 con 0x7f75e40039e0 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f760029e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f75f80b2f20 con 0x7f75f8148530 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f760029e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f75f815e8a0 con 0x7f75f80c9bf0 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75fe03a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 0x7f75f8147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75fe03a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 0x7f75f8147c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51314/0 (socket says 172.21.15.77:51314) 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75fe03a640 1 -- 172.21.15.77:0/1655498449 learned_addr learned my addr 172.21.15.77:0/1655498449 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75fe83b640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f8148530 0x7f75f81489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75fe03a640 1 -- 172.21.15.77:0/1655498449 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80c9bf0 msgr2=0x7f75f80ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75fe03a640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80c9bf0 0x7f75f80ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75fe03a640 1 -- 172.21.15.77:0/1655498449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f8148530 msgr2=0x7f75f81489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75fe03a640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f8148530 0x7f75f81489b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75fe03a640 1 -- 172.21.15.77:0/1655498449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f80b6cb0 con 0x7f75e40039e0 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75fe03a640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 0x7f75f8147c50 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f75f4000f80 tx=0x7f75f4001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75eeffd640 1 -- 172.21.15.77:0/1655498449 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4044070 con 0x7f75e40039e0 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75eeffd640 1 -- 172.21.15.77:0/1655498449 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f402bbd0 con 0x7f75e40039e0 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f760029e640 1 -- 172.21.15.77:0/1655498449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f80b6f90 con 0x7f75e40039e0 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f75eeffd640 1 -- 172.21.15.77:0/1655498449 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4034470 con 0x7f75e40039e0 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.831+0000 7f760029e640 1 -- 172.21.15.77:0/1655498449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f81578a0 con 0x7f75e40039e0 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.832+0000 7f760029e640 1 -- 172.21.15.77:0/1655498449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f8068fb0 con 0x7f75e40039e0 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.833+0000 7f75eeffd640 1 -- 172.21.15.77:0/1655498449 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f75f4047070 con 0x7f75e40039e0 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.833+0000 7f75eeffd640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75dc06be30 0x7f75dc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.833+0000 7f75eeffd640 1 -- 172.21.15.77:0/1655498449 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f75f40aad80 con 0x7f75e40039e0 2024-04-25T01:23:10.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.833+0000 7f75fd839640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75dc06be30 0x7f75dc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.834+0000 7f75fd839640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75dc06be30 0x7f75dc06e2f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f75e8007aa0 tx=0x7f75e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:10.837+0000 7f75eeffd640 1 -- 172.21.15.77:0/1655498449 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f75f4079550 con 0x7f75e40039e0 2024-04-25T01:23:11.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.046+0000 7f760029e640 1 -- 172.21.15.77:0/1655498449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f75f80b7160 con 0x7f75e40039e0 2024-04-25T01:23:11.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.047+0000 7f75eeffd640 1 -- 172.21.15.77:0/1655498449 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+73 (secure 0 0 0) 0x7f75f4078e50 con 0x7f75e40039e0 2024-04-25T01:23:11.027 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T01:23:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.048+0000 7f760029e640 1 -- 172.21.15.77:0/1655498449 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75dc06be30 msgr2=0x7f75dc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.048+0000 7f760029e640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75dc06be30 0x7f75dc06e2f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f75e8007aa0 tx=0x7f75e8009040 comp rx=0 tx=0).stop 2024-04-25T01:23:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.048+0000 7f760029e640 1 -- 172.21.15.77:0/1655498449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 msgr2=0x7f75f8147c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.048+0000 7f760029e640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 0x7f75f8147c50 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f75f4000f80 tx=0x7f75f4001dc0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.048+0000 7f760029e640 1 -- 172.21.15.77:0/1655498449 shutdown_connections 2024-04-25T01:23:11.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.048+0000 7f760029e640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f75dc06be30 0x7f75dc06e2f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.048+0000 7f760029e640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f8148530 0x7f75f81489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.049+0000 7f760029e640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75f80c9bf0 0x7f75f80ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.049+0000 7f760029e640 1 --2- 172.21.15.77:0/1655498449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e40039e0 0x7f75f8147c50 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.049+0000 7f760029e640 1 -- 172.21.15.77:0/1655498449 >> 172.21.15.77:0/1655498449 conn(0x7f75f80d3ef0 msgr2=0x7f75f806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:11.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.049+0000 7f760029e640 1 -- 172.21.15.77:0/1655498449 shutdown_connections 2024-04-25T01:23:11.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.049+0000 7f760029e640 1 -- 172.21.15.77:0/1655498449 wait complete. 2024-04-25T01:23:11.039 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T01:23:11.039 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:23:11.039 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-04-25T01:23:11.040 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:23:11.040 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:11.040 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T01:23:11.040 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T01:23:11.040 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:23:11.040 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:23:11.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:11.040 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-04-25T01:23:11.055 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:23:11.055 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:23:11.056 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42657sec preferred_lft 42657sec 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a9ad:d2f:d456:e65f/64 scope link noprefixroute 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0a:90:ec:3f:d1:57 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::b40c:78ff:fecf:cd0c/64 scope link 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0a:90:ec:3f:d1:57 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:23:11.057 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::890:ecff:fe3f:d157/64 scope link 2024-04-25T01:23:11.058 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:11.058 INFO:teuthology.orchestra.run.smithi077.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:23:11.058 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 12:78:6c:75:3b:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T01:23:11.058 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1078:6cff:fe75:3be8/64 scope link 2024-04-25T01:23:11.058 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:11.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:11.058 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:23:11.118 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-04-25T01:23:11.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:11.120 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-04-25T01:23:11.175 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-04-25T01:23:11.175 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:23:11.175 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:23:11.175 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:23:11.175 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:23:11.176 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:23:11.176 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:23:11.176 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:23:11.176 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:23:11.176 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:23:11.176 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:23:11.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:23:11.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:11.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:11.299 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-04-25T01:23:11.313 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T01:23:11.313 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:23:11.314 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:23:11.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:11.314 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-04-25T01:23:11.377 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:23:11.377 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:11.432 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:23:11.433 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:23:11.433 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:11.487 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:23:11.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:11.487 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:11.568 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:11.568 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:23:11.568 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:23:11.576 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:11.576 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:23:11.576 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:23:11.576 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:23:11.576 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:23:11.581 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:11.581 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:23:11.581 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:23:11.581 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:23:11.581 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:23:11.581 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790" 2024-04-25T01:23:11.581 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790:/ 2024-04-25T01:23:11.582 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=7c3c39ee-077e-476c-8994-eede4fab301d" will pass to kernel 2024-04-25T01:23:11.582 INFO:teuthology.orchestra.run.smithi077.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T01:23:11.582 INFO:teuthology.orchestra.run.smithi077.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:23:11.582 INFO:teuthology.orchestra.run.smithi077.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:23:11.582 INFO:teuthology.orchestra.run.smithi077.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:23:11.582 INFO:teuthology.orchestra.run.smithi077.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:23:11.583 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:23:11.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:11.584 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:11.647 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:23:11.678 INFO:tasks.ceph.mds.c.smithi077.stderr:2024-04-25T01:23:11.697+0000 7f2de6706640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:23:11.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:23:11.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:11.732 INFO:teuthology.orchestra.run.smithi077.stdout:client4877 2024-04-25T01:23:11.732 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:23:11.732 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:11.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:11.747 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls testdir1) 2024-04-25T01:23:11.803 INFO:tasks.ceph.mds.c.smithi077.stderr:2024-04-25T01:23:11.823+0000 7f2de26fe640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x10000000000 2024-04-25T01:23:11.804 INFO:teuthology.orchestra.run.smithi077.stdout:testfile 2024-04-25T01:23:11.804 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:11.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.959+0000 7f8410be2640 1 Processor -- start 2024-04-25T01:23:11.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.959+0000 7f8410be2640 1 -- start start 2024-04-25T01:23:11.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.960+0000 7f8410be2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f840c0cfaf0 0x7f840c0d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.960+0000 7f8410be2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f840c0d1e90 con 0x7f840c0d4a10 2024-04-25T01:23:11.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.960+0000 7f8410be2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f840c0b54a0 con 0x7f840c15e4c0 2024-04-25T01:23:11.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.960+0000 7f8410be2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f840c0d2130 con 0x7f840c0cfaf0 2024-04-25T01:23:11.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.960+0000 7f840ad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f840c0cfaf0 0x7f840c0d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.960+0000 7f840ad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f840c0cfaf0 0x7f840c0d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51316/0 (socket says 172.21.15.77:51316) 2024-04-25T01:23:11.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.960+0000 7f8409d9b640 1 --1- 172.21.15.77:0/985032418 >> v1:172.21.15.77:6789/0 conn(0x7f840c0d4a10 0x7f840c0b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44872/0 (socket says 172.21.15.77:44872) 2024-04-25T01:23:11.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.960+0000 7f840ad9d640 1 -- 172.21.15.77:0/985032418 learned_addr learned my addr 172.21.15.77:0/985032418 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:11.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.961+0000 7f840ad9d640 1 -- 172.21.15.77:0/985032418 >> v1:172.21.15.121:6789/0 conn(0x7f840c15e4c0 legacy=0x7f840c0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:11.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.961+0000 7f840ad9d640 1 -- 172.21.15.77:0/985032418 >> v1:172.21.15.77:6789/0 conn(0x7f840c0d4a10 legacy=0x7f840c0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.961+0000 7f840959a640 1 -- 172.21.15.77:0/985032418 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3142519857 0 0) 0x7f840c0d1e90 con 0x7f840c0d4a10 2024-04-25T01:23:11.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.961+0000 7f840ad9d640 1 -- 172.21.15.77:0/985032418 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840c0b5780 con 0x7f840c0cfaf0 2024-04-25T01:23:11.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.961+0000 7f840ad9d640 1 --2- 172.21.15.77:0/985032418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f840c0cfaf0 0x7f840c0d0900 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8404009870 tx=0x7f840402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ca4d1179ee707 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:11.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.962+0000 7f840959a640 1 -- 172.21.15.77:0/985032418 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404037070 con 0x7f840c0cfaf0 2024-04-25T01:23:11.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.962+0000 7f840959a640 1 -- 172.21.15.77:0/985032418 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8404032440 con 0x7f840c0cfaf0 2024-04-25T01:23:11.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.962+0000 7f840959a640 1 -- 172.21.15.77:0/985032418 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404032810 con 0x7f840c0cfaf0 2024-04-25T01:23:11.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.962+0000 7f8410be2640 1 -- 172.21.15.77:0/985032418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f840c0cfaf0 msgr2=0x7f840c0d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.962+0000 7f8410be2640 1 --2- 172.21.15.77:0/985032418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f840c0cfaf0 0x7f840c0d0900 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8404009870 tx=0x7f840402a210 comp rx=0 tx=0).stop 2024-04-25T01:23:11.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.962+0000 7f8410be2640 1 -- 172.21.15.77:0/985032418 shutdown_connections 2024-04-25T01:23:11.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.962+0000 7f8410be2640 1 --2- 172.21.15.77:0/985032418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f840c0cfaf0 0x7f840c0d0900 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.962+0000 7f8410be2640 1 -- 172.21.15.77:0/985032418 >> 172.21.15.77:0/985032418 conn(0x7f840c0d3ef0 msgr2=0x7f840c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:11.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.962+0000 7f8410be2640 1 -- 172.21.15.77:0/985032418 shutdown_connections 2024-04-25T01:23:11.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.962+0000 7f8410be2640 1 -- 172.21.15.77:0/985032418 wait complete. 2024-04-25T01:23:11.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.964+0000 7f8410be2640 1 Processor -- start 2024-04-25T01:23:11.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.964+0000 7f8410be2640 1 -- start start 2024-04-25T01:23:11.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.964+0000 7f8410be2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f840c0cfaf0 0x7f840c1582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.964+0000 7f8410be2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f840c0d4a10 0x7f840c1f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.964+0000 7f8410be2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f840c15e4c0 0x7f840c15a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.964+0000 7f8410be2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f840c0b30a0 con 0x7f840c15e4c0 2024-04-25T01:23:11.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.964+0000 7f8410be2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f840c0b2f20 con 0x7f840c0d4a10 2024-04-25T01:23:11.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.964+0000 7f8410be2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f840c15e8a0 con 0x7f840c0cfaf0 2024-04-25T01:23:11.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.964+0000 7f840ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f840c15e4c0 0x7f840c15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.964+0000 7f840ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f840c15e4c0 0x7f840c15a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51318/0 (socket says 172.21.15.77:51318) 2024-04-25T01:23:11.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.964+0000 7f840ad9d640 1 -- 172.21.15.77:0/2140972776 learned_addr learned my addr 172.21.15.77:0/2140972776 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f8409d9b640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f840c0d4a10 0x7f840c1f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f840ad9d640 1 -- 172.21.15.77:0/2140972776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f840c0cfaf0 msgr2=0x7f840c1582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f840a59c640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f840c0cfaf0 0x7f840c1582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f840ad9d640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f840c0cfaf0 0x7f840c1582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f840ad9d640 1 -- 172.21.15.77:0/2140972776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f840c0d4a10 msgr2=0x7f840c1f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f840ad9d640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f840c0d4a10 0x7f840c1f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f840ad9d640 1 -- 172.21.15.77:0/2140972776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840c1590d0 con 0x7f840c15e4c0 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f8409d9b640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f840c0d4a10 0x7f840c1f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f840ad9d640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f840c15e4c0 0x7f840c15a730 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f840402a250 tx=0x7f84040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f83faffd640 1 -- 172.21.15.77:0/2140972776 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404041070 con 0x7f840c15e4c0 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f83faffd640 1 -- 172.21.15.77:0/2140972776 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f840403c950 con 0x7f840c15e4c0 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f8410be2640 1 -- 172.21.15.77:0/2140972776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f840c0ccfc0 con 0x7f840c15e4c0 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f83faffd640 1 -- 172.21.15.77:0/2140972776 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404037040 con 0x7f840c15e4c0 2024-04-25T01:23:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.965+0000 7f8410be2640 1 -- 172.21.15.77:0/2140972776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f840c0cd580 con 0x7f840c15e4c0 2024-04-25T01:23:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.966+0000 7f8410be2640 1 -- 172.21.15.77:0/2140972776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f840c068fb0 con 0x7f840c15e4c0 2024-04-25T01:23:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.966+0000 7f83faffd640 1 -- 172.21.15.77:0/2140972776 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8404044070 con 0x7f840c15e4c0 2024-04-25T01:23:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.967+0000 7f83faffd640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f83ec06be30 0x7f83ec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.967+0000 7f840a59c640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f83ec06be30 0x7f83ec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.967+0000 7f83faffd640 1 -- 172.21.15.77:0/2140972776 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f84040abd50 con 0x7f840c15e4c0 2024-04-25T01:23:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.967+0000 7f840a59c640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f83ec06be30 0x7f83ec06e2f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f83fc007fd0 tx=0x7f83fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:11.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:11.971+0000 7f83faffd640 1 -- 172.21.15.77:0/2140972776 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f84040af030 con 0x7f840c15e4c0 2024-04-25T01:23:12.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.180+0000 7f8410be2640 1 -- 172.21.15.77:0/2140972776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f840c15d4b0 con 0x7f840c15e4c0 2024-04-25T01:23:12.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.180+0000 7f83faffd640 1 -- 172.21.15.77:0/2140972776 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5188 (secure 0 0 0) 0x7f840407a640 con 0x7f840c15e4c0 2024-04-25T01:23:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:12.161 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2823778882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2823778882},{"type":"v1","addr":"172.21.15.77:6837","nonce":2823778882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.066906+0000","modified":"2024-04-25T01:23:09.251757+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5355":{"gid":5355,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6835/3139470828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3139470828},{"type":"v1","addr":"172.21.15.77:6835","nonce":3139470828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5355,"qdb_cluster":[5355]},"id":2}]} 2024-04-25T01:23:12.161 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-04-25T01:23:12.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.182+0000 7f8410be2640 1 -- 172.21.15.77:0/2140972776 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f83ec06be30 msgr2=0x7f83ec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.182+0000 7f8410be2640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f83ec06be30 0x7f83ec06e2f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f83fc007fd0 tx=0x7f83fc009040 comp rx=0 tx=0).stop 2024-04-25T01:23:12.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.182+0000 7f8410be2640 1 -- 172.21.15.77:0/2140972776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f840c15e4c0 msgr2=0x7f840c15a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.183+0000 7f8410be2640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f840c15e4c0 0x7f840c15a730 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f840402a250 tx=0x7f84040046c0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.183+0000 7f8410be2640 1 -- 172.21.15.77:0/2140972776 shutdown_connections 2024-04-25T01:23:12.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.183+0000 7f8410be2640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f83ec06be30 0x7f83ec06e2f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.183+0000 7f8410be2640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f840c15e4c0 0x7f840c15a730 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.183+0000 7f8410be2640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f840c0d4a10 0x7f840c1f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.183+0000 7f8410be2640 1 --2- 172.21.15.77:0/2140972776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f840c0cfaf0 0x7f840c1582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.183+0000 7f8410be2640 1 -- 172.21.15.77:0/2140972776 >> 172.21.15.77:0/2140972776 conn(0x7f840c0d3ef0 msgr2=0x7f840c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.183+0000 7f8410be2640 1 -- 172.21.15.77:0/2140972776 shutdown_connections 2024-04-25T01:23:12.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.183+0000 7f8410be2640 1 -- 172.21.15.77:0/2140972776 wait complete. 2024-04-25T01:23:12.174 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-04-25T01:23:12.298 INFO:teuthology.orchestra.run.smithi077.stdout:{"message":"","return_code":0} 2024-04-25T01:23:12.298 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-25T01:23:12.298 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000000.00000000 parent 2024-04-25T01:23:12.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f711b51cf40 1 Processor -- start 2024-04-25T01:23:12.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f711b51cf40 1 -- start start 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f711b51cf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5638ebbe52d0 0x5638ebbe9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f711b51cf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5638ebc8bdb0 0x5638ebc8c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f711b51cf40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbb8520 0x5638ebd239f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f711b51cf40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5638ebb61640 con 0x5638ebbb8520 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f711b51cf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5638ebb65680 con 0x5638ebbe52d0 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f711b51cf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5638ebba7f50 con 0x5638ebc8bdb0 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f7118dba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbb8520 0x5638ebd239f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f7118dba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbb8520 0x5638ebd239f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51336/0 (socket says 172.21.15.77:51336) 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f7118dba640 1 -- 172.21.15.77:0/3788665390 learned_addr learned my addr 172.21.15.77:0/3788665390 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.360+0000 7f71185b9640 1 --2- 172.21.15.77:0/3788665390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5638ebbe52d0 0x5638ebbe9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f7117db8640 1 --2- 172.21.15.77:0/3788665390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5638ebc8bdb0 0x5638ebc8c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f7118dba640 1 -- 172.21.15.77:0/3788665390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5638ebc8bdb0 msgr2=0x5638ebc8c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:12.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f7118dba640 1 --2- 172.21.15.77:0/3788665390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5638ebc8bdb0 0x5638ebc8c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f7118dba640 1 -- 172.21.15.77:0/3788665390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5638ebbe52d0 msgr2=0x5638ebbe9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f7118dba640 1 --2- 172.21.15.77:0/3788665390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5638ebbe52d0 0x5638ebbe9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f7118dba640 1 -- 172.21.15.77:0/3788665390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638ebbe8ce0 con 0x5638ebbb8520 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f71185b9640 1 --2- 172.21.15.77:0/3788665390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5638ebbe52d0 0x5638ebbe9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f7118dba640 1 --2- 172.21.15.77:0/3788665390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbb8520 0x5638ebd239f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7110002d60 tx=0x7f711000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f77259910c9784 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f7116db6640 1 -- 172.21.15.77:0/3788665390 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110013020 con 0x5638ebbb8520 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f7116db6640 1 -- 172.21.15.77:0/3788665390 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f711000f400 con 0x5638ebbb8520 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f7116db6640 1 -- 172.21.15.77:0/3788665390 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711000f7f0 con 0x5638ebbb8520 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f711b51cf40 1 -- 172.21.15.77:0/3788665390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbb8520 msgr2=0x5638ebd239f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f711b51cf40 1 --2- 172.21.15.77:0/3788665390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbb8520 0x5638ebd239f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7110002d60 tx=0x7f711000b5b0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f711b51cf40 1 -- 172.21.15.77:0/3788665390 shutdown_connections 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f711b51cf40 1 --2- 172.21.15.77:0/3788665390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbb8520 0x5638ebd239f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f711b51cf40 1 --2- 172.21.15.77:0/3788665390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5638ebc8bdb0 0x5638ebc8c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f711b51cf40 1 --2- 172.21.15.77:0/3788665390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5638ebbe52d0 0x5638ebbe9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f711b51cf40 1 -- 172.21.15.77:0/3788665390 >> 172.21.15.77:0/3788665390 conn(0x5638ebc00b60 msgr2=0x5638ebbe9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f711b51cf40 1 -- 172.21.15.77:0/3788665390 shutdown_connections 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.361+0000 7f711b51cf40 1 -- 172.21.15.77:0/3788665390 wait complete. 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.362+0000 7f711b51cf40 1 Processor -- start 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.362+0000 7f711b51cf40 1 -- start start 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f711b51cf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5638ebbb8520 0x5638ebc857c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f711b51cf40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbe52d0 0x5638ebd21720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f711b51cf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5638ebc8bdb0 0x5638ebc84c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f711b51cf40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5638ebc89f90 con 0x5638ebbe52d0 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f711b51cf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5638ebc8c880 con 0x5638ebbb8520 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f711b51cf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5638ebc89960 con 0x5638ebc8bdb0 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7117db8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbe52d0 0x5638ebd21720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7117db8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbe52d0 0x5638ebd21720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51348/0 (socket says 172.21.15.77:51348) 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7117db8640 1 -- 172.21.15.77:0/2981700900 learned_addr learned my addr 172.21.15.77:0/2981700900 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7118dba640 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5638ebc8bdb0 0x5638ebc84c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f71185b9640 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5638ebbb8520 0x5638ebc857c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7117db8640 1 -- 172.21.15.77:0/2981700900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5638ebc8bdb0 msgr2=0x5638ebc84c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7117db8640 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5638ebc8bdb0 0x5638ebc84c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7117db8640 1 -- 172.21.15.77:0/2981700900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5638ebbb8520 msgr2=0x5638ebc857c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7117db8640 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5638ebbb8520 0x5638ebc857c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7117db8640 1 -- 172.21.15.77:0/2981700900 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638ebbe94f0 con 0x5638ebbe52d0 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7117db8640 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbe52d0 0x5638ebd21720 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f710c057040 tx=0x7f710c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7115db4640 1 -- 172.21.15.77:0/2981700900 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c063030 con 0x5638ebbe52d0 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7115db4640 1 -- 172.21.15.77:0/2981700900 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f710c0548b0 con 0x5638ebbe52d0 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f711b51cf40 1 -- 172.21.15.77:0/2981700900 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5638ebbe8ee0 con 0x5638ebbe52d0 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f7115db4640 1 -- 172.21.15.77:0/2981700900 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c059040 con 0x5638ebbe52d0 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.363+0000 7f711b51cf40 1 -- 172.21.15.77:0/2981700900 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638ebbe87c0 con 0x5638ebbe52d0 2024-04-25T01:23:12.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.365+0000 7f7115db4640 1 -- 172.21.15.77:0/2981700900 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f710c066030 con 0x5638ebbe52d0 2024-04-25T01:23:12.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.365+0000 7f7115db4640 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f710406a810 0x7f710406ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.365+0000 7f7115db4640 1 -- 172.21.15.77:0/2981700900 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f710c0cd550 con 0x5638ebbe52d0 2024-04-25T01:23:12.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.365+0000 7f71185b9640 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f710406a810 0x7f710406ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.365+0000 7f711b51cf40 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x5638ebd21c60 0x5638ebc8e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.365+0000 7f711b51cf40 1 -- 172.21.15.77:0/2981700900 --> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] -- osd_op(unknown.0.0:1 5.10 5:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x5638ebc89290 con 0x5638ebd21c60 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.365+0000 7f71185b9640 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f710406a810 0x7f710406ccd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f71080044f0 tx=0x7f7108008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.365+0000 7f7118dba640 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x5638ebd21c60 0x5638ebc8e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f7118dba640 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x5638ebd21c60 0x5638ebc8e700 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f7118dba640 1 -- 172.21.15.77:0/2981700900 <== osd.7 v2:172.21.15.121:6800/251765690 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (64) out=64b] v0'0 uv17 ondisk = 0) ==== 164+0+64 (crc 0 0 0) 0x7f711000f5b0 con 0x5638ebd21c60 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f711b51cf40 1 -- 172.21.15.77:0/2981700900 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x5638ebd21c60 msgr2=0x5638ebc8e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f711b51cf40 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x5638ebd21c60 0x5638ebc8e700 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f711b51cf40 1 -- 172.21.15.77:0/2981700900 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f710406a810 msgr2=0x7f710406ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f711b51cf40 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f710406a810 0x7f710406ccd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f71080044f0 tx=0x7f7108008040 comp rx=0 tx=0).stop 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f711b51cf40 1 -- 172.21.15.77:0/2981700900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbe52d0 msgr2=0x5638ebd21720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f711b51cf40 1 --2- 172.21.15.77:0/2981700900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638ebbe52d0 0x5638ebd21720 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f710c057040 tx=0x7f710c002fb0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f7118dba640 1 -- 172.21.15.77:0/2981700900 reap_dead start 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f711b51cf40 1 -- 172.21.15.77:0/2981700900 shutdown_connections 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f711b51cf40 1 -- 172.21.15.77:0/2981700900 >> 172.21.15.77:0/2981700900 conn(0x5638ebc00b60 msgr2=0x5638ebbe9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f711b51cf40 1 -- 172.21.15.77:0/2981700900 shutdown_connections 2024-04-25T01:23:12.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.366+0000 7f711b51cf40 1 -- 172.21.15.77:0/2981700900 wait complete. 2024-04-25T01:23:12.352 INFO:teuthology.orchestra.run.smithi077.stdout::testdir1 2024-04-25T01:23:12.352 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T01:23:12.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:23:12.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:23:12.480 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T01:23:12.480 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:23:12.481 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:23:12.481 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:12.496 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:23:12.496 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:12.551 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:12.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.707+0000 7fa908238640 1 Processor -- start 2024-04-25T01:23:12.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.707+0000 7fa908238640 1 -- start start 2024-04-25T01:23:12.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa908238640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa900154ac0 0x7fa90015b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa908238640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9000cf2a0 0x7fa9000711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa908238640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa900071730 0x7fa90006e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa908238640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fa90006d540 con 0x7fa900154ac0 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa908238640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa900154ea0 con 0x7fa900071730 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa908238640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa90006d790 con 0x7fa9000cf2a0 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa905fd4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa900154ac0 0x7fa90015b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa905fd4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa900154ac0 0x7fa90015b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51364/0 (socket says 172.21.15.77:51364) 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa905fd4640 1 -- 172.21.15.77:0/1727473003 learned_addr learned my addr 172.21.15.77:0/1727473003 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa9067d5640 1 --2- 172.21.15.77:0/1727473003 >> v2:172.21.15.121:3300/0 conn(0x7fa900071730 0x7fa90006e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa9057d3640 1 --2- 172.21.15.77:0/1727473003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9000cf2a0 0x7fa9000711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa905fd4640 1 -- 172.21.15.77:0/1727473003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9000cf2a0 msgr2=0x7fa9000711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa905fd4640 1 --2- 172.21.15.77:0/1727473003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9000cf2a0 0x7fa9000711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa905fd4640 1 -- 172.21.15.77:0/1727473003 >> v2:172.21.15.121:3300/0 conn(0x7fa900071730 msgr2=0x7fa90006e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa905fd4640 1 --2- 172.21.15.77:0/1727473003 >> v2:172.21.15.121:3300/0 conn(0x7fa900071730 0x7fa90006e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa905fd4640 1 -- 172.21.15.77:0/1727473003 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90006da70 con 0x7fa900154ac0 2024-04-25T01:23:12.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.708+0000 7fa9067d5640 1 --2- 172.21.15.77:0/1727473003 >> v2:172.21.15.121:3300/0 conn(0x7fa900071730 0x7fa90006e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:12.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.709+0000 7fa905fd4640 1 --2- 172.21.15.77:0/1727473003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa900154ac0 0x7fa90015b5f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa900073970 tx=0x7fa8fc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73d2ec6c6a32a44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.709+0000 7fa904fd2640 1 -- 172.21.15.77:0/1727473003 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc037070 con 0x7fa900154ac0 2024-04-25T01:23:12.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.709+0000 7fa904fd2640 1 -- 172.21.15.77:0/1727473003 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8fc032440 con 0x7fa900154ac0 2024-04-25T01:23:12.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.709+0000 7fa904fd2640 1 -- 172.21.15.77:0/1727473003 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc032810 con 0x7fa900154ac0 2024-04-25T01:23:12.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.710+0000 7fa908238640 1 -- 172.21.15.77:0/1727473003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa900154ac0 msgr2=0x7fa90015b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.710+0000 7fa908238640 1 --2- 172.21.15.77:0/1727473003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa900154ac0 0x7fa90015b5f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa900073970 tx=0x7fa8fc02a1e0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.710+0000 7fa908238640 1 -- 172.21.15.77:0/1727473003 shutdown_connections 2024-04-25T01:23:12.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.710+0000 7fa908238640 1 --2- 172.21.15.77:0/1727473003 >> v2:172.21.15.121:3300/0 conn(0x7fa900071730 0x7fa90006e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.710+0000 7fa908238640 1 --2- 172.21.15.77:0/1727473003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9000cf2a0 0x7fa9000711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.710+0000 7fa908238640 1 --2- 172.21.15.77:0/1727473003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa900154ac0 0x7fa90015b5f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.710+0000 7fa908238640 1 -- 172.21.15.77:0/1727473003 >> 172.21.15.77:0/1727473003 conn(0x7fa9000b1820 msgr2=0x7fa9000b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.710+0000 7fa908238640 1 -- 172.21.15.77:0/1727473003 shutdown_connections 2024-04-25T01:23:12.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.710+0000 7fa908238640 1 -- 172.21.15.77:0/1727473003 wait complete. 2024-04-25T01:23:12.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.711+0000 7fa908238640 1 Processor -- start 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.711+0000 7fa908238640 1 -- start start 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa908238640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa900071730 0x7fa900157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa908238640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9000cf2a0 0x7fa90014a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa908238640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa900154ac0 0x7fa900142d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa908238640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa9000cf720 con 0x7fa900154ac0 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa908238640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa90006e030 con 0x7fa900071730 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa908238640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa90006ea90 con 0x7fa9000cf2a0 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa9067d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa900154ac0 0x7fa900142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa9067d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa900154ac0 0x7fa900142d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51374/0 (socket says 172.21.15.77:51374) 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa9067d5640 1 -- 172.21.15.77:0/3229441656 learned_addr learned my addr 172.21.15.77:0/3229441656 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa9067d5640 1 -- 172.21.15.77:0/3229441656 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9000cf2a0 msgr2=0x7fa90014a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa9067d5640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9000cf2a0 0x7fa90014a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa9067d5640 1 -- 172.21.15.77:0/3229441656 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa900071730 msgr2=0x7fa900157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa905fd4640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa900071730 0x7fa900157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa9067d5640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa900071730 0x7fa900157be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa9067d5640 1 -- 172.21.15.77:0/3229441656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90014a8b0 con 0x7fa900154ac0 2024-04-25T01:23:12.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa9067d5640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa900154ac0 0x7fa900142d50 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800cef0 tx=0x7fa8f800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa8f67fc640 1 -- 172.21.15.77:0/3229441656 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f801a070 con 0x7fa900154ac0 2024-04-25T01:23:12.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa8f67fc640 1 -- 172.21.15.77:0/3229441656 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8f800d840 con 0x7fa900154ac0 2024-04-25T01:23:12.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa908238640 1 -- 172.21.15.77:0/3229441656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa900158250 con 0x7fa900154ac0 2024-04-25T01:23:12.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa8f67fc640 1 -- 172.21.15.77:0/3229441656 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f800dc30 con 0x7fa900154ac0 2024-04-25T01:23:12.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.712+0000 7fa908238640 1 -- 172.21.15.77:0/3229441656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa900142550 con 0x7fa900154ac0 2024-04-25T01:23:12.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.713+0000 7fa908238640 1 -- 172.21.15.77:0/3229441656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa900142080 con 0x7fa900154ac0 2024-04-25T01:23:12.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.714+0000 7fa8f67fc640 1 -- 172.21.15.77:0/3229441656 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa8f801d070 con 0x7fa900154ac0 2024-04-25T01:23:12.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.714+0000 7fa8f67fc640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8e406be30 0x7fa8e406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.714+0000 7fa905fd4640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8e406be30 0x7fa8e406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.714+0000 7fa8f67fc640 1 -- 172.21.15.77:0/3229441656 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7fa8f808b790 con 0x7fa900154ac0 2024-04-25T01:23:12.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.715+0000 7fa905fd4640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8e406be30 0x7fa8e406e2f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc034000 tx=0x7fa8fc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.718+0000 7fa8f67fc640 1 -- 172.21.15.77:0/3229441656 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa8f8059ff0 con 0x7fa900154ac0 2024-04-25T01:23:12.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.895+0000 7fa908238640 1 -- 172.21.15.77:0/3229441656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa900068fe0 con 0x7fa900154ac0 2024-04-25T01:23:12.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.897+0000 7fa8f67fc640 1 -- 172.21.15.77:0/3229441656 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+17071 (secure 0 0 0) 0x7fa900068fe0 con 0x7fa900154ac0 2024-04-25T01:23:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:12.877 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":39,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:23:06.238842+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.457887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.029883+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6837/2944320661":"2024-04-26T01:23:06.238820+0000","172.21.15.77:6836/2944320661":"2024-04-26T01:23:06.238820+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:12.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.898+0000 7fa908238640 1 -- 172.21.15.77:0/3229441656 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8e406be30 msgr2=0x7fa8e406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.898+0000 7fa908238640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8e406be30 0x7fa8e406e2f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc034000 tx=0x7fa8fc02e000 comp rx=0 tx=0).stop 2024-04-25T01:23:12.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.899+0000 7fa908238640 1 -- 172.21.15.77:0/3229441656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa900154ac0 msgr2=0x7fa900142d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.899+0000 7fa908238640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa900154ac0 0x7fa900142d50 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800cef0 tx=0x7fa8f800a710 comp rx=0 tx=0).stop 2024-04-25T01:23:12.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.899+0000 7fa908238640 1 -- 172.21.15.77:0/3229441656 shutdown_connections 2024-04-25T01:23:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.899+0000 7fa908238640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fa8e406be30 0x7fa8e406e2f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.899+0000 7fa908238640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa900154ac0 0x7fa900142d50 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.899+0000 7fa908238640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9000cf2a0 0x7fa90014a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.899+0000 7fa908238640 1 --2- 172.21.15.77:0/3229441656 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa900071730 0x7fa900157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.899+0000 7fa908238640 1 -- 172.21.15.77:0/3229441656 >> 172.21.15.77:0/3229441656 conn(0x7fa9000b1820 msgr2=0x7fa90014d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.899+0000 7fa908238640 1 -- 172.21.15.77:0/3229441656 shutdown_connections 2024-04-25T01:23:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:12.899+0000 7fa908238640 1 -- 172.21.15.77:0/3229441656 wait complete. 2024-04-25T01:23:12.890 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.457887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.029883+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:23:12.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:23:12.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.009+0000 7f02d58a7640 1 Processor -- start 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.009+0000 7f02d58a7640 1 -- start start 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02d58a7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f02d00720c0 0x7f02d00b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02d58a7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02d006caf0 0x7f02d006d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02d58a7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02d015b6a0 0x7f02d006d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02d58a7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f02d015bb80 con 0x7f02d006caf0 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02d58a7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f02d00724a0 con 0x7f02d00720c0 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02d58a7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f02d00b3750 con 0x7f02d015b6a0 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02ce7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02d006caf0 0x7f02d006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02ce7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02d006caf0 0x7f02d006d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51378/0 (socket says 172.21.15.77:51378) 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02ce7fc640 1 -- 172.21.15.77:0/1018430122 learned_addr learned my addr 172.21.15.77:0/1018430122 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02ce7fc640 1 -- 172.21.15.77:0/1018430122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02d015b6a0 msgr2=0x7f02d006d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:12.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02ce7fc640 1 --2- 172.21.15.77:0/1018430122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02d015b6a0 0x7f02d006d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02ce7fc640 1 -- 172.21.15.77:0/1018430122 >> v2:172.21.15.121:3300/0 conn(0x7f02d00720c0 msgr2=0x7f02d00b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:12.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02ce7fc640 1 --2- 172.21.15.77:0/1018430122 >> v2:172.21.15.121:3300/0 conn(0x7f02d00720c0 0x7f02d00b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.010+0000 7f02ce7fc640 1 -- 172.21.15.77:0/1018430122 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d00b3a30 con 0x7f02d006caf0 2024-04-25T01:23:12.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.011+0000 7f02ce7fc640 1 --2- 172.21.15.77:0/1018430122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d006caf0 0x7f02d006d370 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f02d00b3130 tx=0x7f02c402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3415b74c2be143 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.011+0000 7f02cdffb640 1 -- 172.21.15.77:0/1018430122 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4037070 con 0x7f02d006caf0 2024-04-25T01:23:12.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.011+0000 7f02cdffb640 1 -- 172.21.15.77:0/1018430122 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02c4032440 con 0x7f02d006caf0 2024-04-25T01:23:12.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.011+0000 7f02cdffb640 1 -- 172.21.15.77:0/1018430122 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4032810 con 0x7f02d006caf0 2024-04-25T01:23:12.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.012+0000 7f02d58a7640 1 -- 172.21.15.77:0/1018430122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d006caf0 msgr2=0x7f02d006d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.012+0000 7f02d58a7640 1 --2- 172.21.15.77:0/1018430122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d006caf0 0x7f02d006d370 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f02d00b3130 tx=0x7f02c402a180 comp rx=0 tx=0).stop 2024-04-25T01:23:12.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.012+0000 7f02d58a7640 1 -- 172.21.15.77:0/1018430122 shutdown_connections 2024-04-25T01:23:12.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.012+0000 7f02d58a7640 1 --2- 172.21.15.77:0/1018430122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02d015b6a0 0x7f02d006d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.012+0000 7f02d58a7640 1 --2- 172.21.15.77:0/1018430122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d006caf0 0x7f02d006d370 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.012+0000 7f02d58a7640 1 --2- 172.21.15.77:0/1018430122 >> v2:172.21.15.121:3300/0 conn(0x7f02d00720c0 0x7f02d00b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.012+0000 7f02d58a7640 1 -- 172.21.15.77:0/1018430122 >> 172.21.15.77:0/1018430122 conn(0x7f02d006b720 msgr2=0x7f02d00b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.012+0000 7f02d58a7640 1 -- 172.21.15.77:0/1018430122 shutdown_connections 2024-04-25T01:23:12.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.012+0000 7f02d58a7640 1 -- 172.21.15.77:0/1018430122 wait complete. 2024-04-25T01:23:12.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.013+0000 7f02d58a7640 1 Processor -- start 2024-04-25T01:23:12.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.013+0000 7f02d58a7640 1 -- start start 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02d58a7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02d006caf0 0x7f02d0146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02d58a7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d00720c0 0x7f02d00b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02d58a7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02d015b6a0 0x7f02d00b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02d58a7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f02d00cf3c0 con 0x7f02d00720c0 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02d58a7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f02d00b3ff0 con 0x7f02d006caf0 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02d58a7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f02d00cf540 con 0x7f02d015b6a0 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02ce7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d00720c0 0x7f02d00b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02ce7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d00720c0 0x7f02d00b0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51384/0 (socket says 172.21.15.77:51384) 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02ce7fc640 1 -- 172.21.15.77:0/4083332732 learned_addr learned my addr 172.21.15.77:0/4083332732 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02ce7fc640 1 -- 172.21.15.77:0/4083332732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02d015b6a0 msgr2=0x7f02d00b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02ce7fc640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02d015b6a0 0x7f02d00b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02ce7fc640 1 -- 172.21.15.77:0/4083332732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02d006caf0 msgr2=0x7f02d0146660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02ce7fc640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02d006caf0 0x7f02d0146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02ce7fc640 1 -- 172.21.15.77:0/4083332732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d014a530 con 0x7f02d00720c0 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02ce7fc640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d00720c0 0x7f02d00b0830 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f02c40097f0 tx=0x7f02c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02b7fff640 1 -- 172.21.15.77:0/4083332732 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4041070 con 0x7f02d00720c0 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02b7fff640 1 -- 172.21.15.77:0/4083332732 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02c403c870 con 0x7f02d00720c0 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.014+0000 7f02d58a7640 1 -- 172.21.15.77:0/4083332732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02d014a810 con 0x7f02d00720c0 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.015+0000 7f02b7fff640 1 -- 172.21.15.77:0/4083332732 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4037040 con 0x7f02d00720c0 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.015+0000 7f02d58a7640 1 -- 172.21.15.77:0/4083332732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02d014add0 con 0x7f02d00720c0 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.015+0000 7f02d58a7640 1 -- 172.21.15.77:0/4083332732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02d0068fb0 con 0x7f02d00720c0 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.016+0000 7f02b7fff640 1 -- 172.21.15.77:0/4083332732 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f02c4044070 con 0x7f02d00720c0 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.016+0000 7f02b7fff640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f02bc06be30 0x7f02bc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.016+0000 7f02b7fff640 1 -- 172.21.15.77:0/4083332732 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f02c40abcc0 con 0x7f02d00720c0 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.016+0000 7f02ceffd640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f02bc06be30 0x7f02bc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.017+0000 7f02ceffd640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f02bc06be30 0x7f02bc06e2f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f02d0154b80 tx=0x7f02c0009250 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.021+0000 7f02b7fff640 1 -- 172.21.15.77:0/4083332732 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f02c407a640 con 0x7f02d00720c0 2024-04-25T01:23:13.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.196+0000 7f02d58a7640 1 -- 172.21.15.77:0/4083332732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f02d00b3400 con 0x7f02d00720c0 2024-04-25T01:23:13.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.196+0000 7f02b7fff640 1 -- 172.21.15.77:0/4083332732 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f02c4079f40 con 0x7f02d00720c0 2024-04-25T01:23:13.177 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:23:13.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.198+0000 7f02d58a7640 1 -- 172.21.15.77:0/4083332732 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f02bc06be30 msgr2=0x7f02bc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.198+0000 7f02d58a7640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f02bc06be30 0x7f02bc06e2f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f02d0154b80 tx=0x7f02c0009250 comp rx=0 tx=0).stop 2024-04-25T01:23:13.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.198+0000 7f02d58a7640 1 -- 172.21.15.77:0/4083332732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d00720c0 msgr2=0x7f02d00b0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.198+0000 7f02d58a7640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d00720c0 0x7f02d00b0830 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f02c40097f0 tx=0x7f02c40046c0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.199+0000 7f02d58a7640 1 -- 172.21.15.77:0/4083332732 shutdown_connections 2024-04-25T01:23:13.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.199+0000 7f02d58a7640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f02bc06be30 0x7f02bc06e2f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.199+0000 7f02d58a7640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d00720c0 0x7f02d00b0830 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.199+0000 7f02d58a7640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02d015b6a0 0x7f02d00b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.199+0000 7f02d58a7640 1 --2- 172.21.15.77:0/4083332732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02d006caf0 0x7f02d0146660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.199+0000 7f02d58a7640 1 -- 172.21.15.77:0/4083332732 >> 172.21.15.77:0/4083332732 conn(0x7f02d006b720 msgr2=0x7f02d015ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:13.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.199+0000 7f02d58a7640 1 -- 172.21.15.77:0/4083332732 shutdown_connections 2024-04-25T01:23:13.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.199+0000 7f02d58a7640 1 -- 172.21.15.77:0/4083332732 wait complete. 2024-04-25T01:23:13.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:23:13.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.310+0000 7f2459ac3640 1 Processor -- start 2024-04-25T01:23:13.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.310+0000 7f2459ac3640 1 -- start start 2024-04-25T01:23:13.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.310+0000 7f2459ac3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f24540d23d0 0x7f24540d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.310+0000 7f2459ac3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f24540d27b0 0x7f24540d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.310+0000 7f2459ac3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24540b9380 0x7f24540b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.310+0000 7f2459ac3640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f24540cea40 con 0x7f24540d27b0 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.310+0000 7f2459ac3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f24540b9d10 con 0x7f24540d23d0 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.310+0000 7f2459ac3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f24540cece0 con 0x7f24540b9380 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f2452ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f24540d27b0 0x7f24540d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f2452ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f24540d27b0 0x7f24540d0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51388/0 (socket says 172.21.15.77:51388) 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f2452ffd640 1 -- 172.21.15.77:0/4034349588 learned_addr learned my addr 172.21.15.77:0/4034349588 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f2453fff640 1 --2- 172.21.15.77:0/4034349588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24540b9380 0x7f24540b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f24537fe640 1 --2- 172.21.15.77:0/4034349588 >> v2:172.21.15.121:3300/0 conn(0x7f24540d23d0 0x7f24540d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f2452ffd640 1 -- 172.21.15.77:0/4034349588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24540b9380 msgr2=0x7f24540b97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f2452ffd640 1 --2- 172.21.15.77:0/4034349588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24540b9380 0x7f24540b97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f2452ffd640 1 -- 172.21.15.77:0/4034349588 >> v2:172.21.15.121:3300/0 conn(0x7f24540d23d0 msgr2=0x7f24540d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f2452ffd640 1 --2- 172.21.15.77:0/4034349588 >> v2:172.21.15.121:3300/0 conn(0x7f24540d23d0 0x7f24540d4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f2452ffd640 1 -- 172.21.15.77:0/4034349588 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24540cefc0 con 0x7f24540d27b0 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f24537fe640 1 --2- 172.21.15.77:0/4034349588 >> v2:172.21.15.121:3300/0 conn(0x7f24540d23d0 0x7f24540d4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.311+0000 7f2452ffd640 1 --2- 172.21.15.77:0/4034349588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24540d27b0 0x7f24540d0ee0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2454067db0 tx=0x7f244802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1092b53bdd5f99a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f24527fc640 1 -- 172.21.15.77:0/4034349588 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448037070 con 0x7f24540d27b0 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f24527fc640 1 -- 172.21.15.77:0/4034349588 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2448032440 con 0x7f24540d27b0 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f24527fc640 1 -- 172.21.15.77:0/4034349588 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448032810 con 0x7f24540d27b0 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f2459ac3640 1 -- 172.21.15.77:0/4034349588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24540d27b0 msgr2=0x7f24540d0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f2459ac3640 1 --2- 172.21.15.77:0/4034349588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24540d27b0 0x7f24540d0ee0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2454067db0 tx=0x7f244802a270 comp rx=0 tx=0).stop 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f2459ac3640 1 -- 172.21.15.77:0/4034349588 shutdown_connections 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f2459ac3640 1 --2- 172.21.15.77:0/4034349588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24540b9380 0x7f24540b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f2459ac3640 1 --2- 172.21.15.77:0/4034349588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24540d27b0 0x7f24540d0ee0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f2459ac3640 1 --2- 172.21.15.77:0/4034349588 >> v2:172.21.15.121:3300/0 conn(0x7f24540d23d0 0x7f24540d4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f2459ac3640 1 -- 172.21.15.77:0/4034349588 >> 172.21.15.77:0/4034349588 conn(0x7f24540d3e30 msgr2=0x7f24540d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:13.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f2459ac3640 1 -- 172.21.15.77:0/4034349588 shutdown_connections 2024-04-25T01:23:13.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.312+0000 7f2459ac3640 1 -- 172.21.15.77:0/4034349588 wait complete. 2024-04-25T01:23:13.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.313+0000 7f2459ac3640 1 Processor -- start 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f2459ac3640 1 -- start start 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f2459ac3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24540b9380 0x7f245414fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f2459ac3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24540d23d0 0x7f24540b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f2459ac3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24540d27b0 0x7f24540c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f2459ac3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f24540cf860 con 0x7f24540d27b0 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f2459ac3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f24540cf580 con 0x7f24540d23d0 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f2459ac3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f24540cfd80 con 0x7f24540b9380 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f2453fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24540d27b0 0x7f24540c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f2453fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24540d27b0 0x7f24540c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51390/0 (socket says 172.21.15.77:51390) 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f2453fff640 1 -- 172.21.15.77:0/4196311307 learned_addr learned my addr 172.21.15.77:0/4196311307 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f24537fe640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24540b9380 0x7f245414fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.314+0000 7f2452ffd640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24540d23d0 0x7f24540b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f2453fff640 1 -- 172.21.15.77:0/4196311307 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24540b9380 msgr2=0x7f245414fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f2453fff640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24540b9380 0x7f245414fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f2453fff640 1 -- 172.21.15.77:0/4196311307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24540d23d0 msgr2=0x7f24540b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f2453fff640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24540d23d0 0x7f24540b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f2453fff640 1 -- 172.21.15.77:0/4196311307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2454150380 con 0x7f24540d27b0 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f24537fe640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24540b9380 0x7f245414fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f2453fff640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24540d27b0 0x7f24540c9d10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f244c007e60 tx=0x7f244c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f243bfff640 1 -- 172.21.15.77:0/4196311307 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c014620 con 0x7f24540d27b0 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f243bfff640 1 -- 172.21.15.77:0/4196311307 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f244c019070 con 0x7f24540d27b0 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f2459ac3640 1 -- 172.21.15.77:0/4196311307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2454150a40 con 0x7f24540d27b0 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f243bfff640 1 -- 172.21.15.77:0/4196311307 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c014a20 con 0x7f24540d27b0 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.315+0000 7f2459ac3640 1 -- 172.21.15.77:0/4196311307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2454151000 con 0x7f24540d27b0 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.316+0000 7f2459ac3640 1 -- 172.21.15.77:0/4196311307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2454068fb0 con 0x7f24540d27b0 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.316+0000 7f243bfff640 1 -- 172.21.15.77:0/4196311307 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f244c014bc0 con 0x7f24540d27b0 2024-04-25T01:23:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.317+0000 7f243bfff640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f244006be30 0x7f244006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.317+0000 7f243bfff640 1 -- 172.21.15.77:0/4196311307 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f244c08b790 con 0x7f24540d27b0 2024-04-25T01:23:13.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.317+0000 7f24537fe640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f244006be30 0x7f244006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.317+0000 7f24537fe640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f244006be30 0x7f244006e2f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f244400aa30 tx=0x7f2444008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.321+0000 7f243bfff640 1 -- 172.21.15.77:0/4196311307 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2454150380 con 0x7f24540d27b0 2024-04-25T01:23:13.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.492+0000 7f2459ac3640 1 -- 172.21.15.77:0/4196311307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f245406d720 con 0x7f24540d27b0 2024-04-25T01:23:13.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.493+0000 7f243bfff640 1 -- 172.21.15.77:0/4196311307 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f244c059ff0 con 0x7f24540d27b0 2024-04-25T01:23:13.473 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:23:13.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 -- 172.21.15.77:0/4196311307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f244006be30 msgr2=0x7f244006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f244006be30 0x7f244006e2f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f244400aa30 tx=0x7f2444008040 comp rx=0 tx=0).stop 2024-04-25T01:23:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 -- 172.21.15.77:0/4196311307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24540d27b0 msgr2=0x7f24540c9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24540d27b0 0x7f24540c9d10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f244c007e60 tx=0x7f244c00e4b0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 -- 172.21.15.77:0/4196311307 shutdown_connections 2024-04-25T01:23:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f244006be30 0x7f244006e2f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24540d27b0 0x7f24540c9d10 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24540d23d0 0x7f24540b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 --2- 172.21.15.77:0/4196311307 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24540b9380 0x7f245414fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 -- 172.21.15.77:0/4196311307 >> 172.21.15.77:0/4196311307 conn(0x7f24540d3e30 msgr2=0x7f24540b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 -- 172.21.15.77:0/4196311307 shutdown_connections 2024-04-25T01:23:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.495+0000 7f2459ac3640 1 -- 172.21.15.77:0/4196311307 wait complete. 2024-04-25T01:23:13.486 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:23:13.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.605+0000 7f86e87d5640 1 Processor -- start 2024-04-25T01:23:13.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.605+0000 7f86e87d5640 1 -- start start 2024-04-25T01:23:13.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e87d5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86e015c0f0 0x7f86e00b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e87d5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f86e00b2a20 0x7f86e00b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e87d5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e015c4d0 con 0x7f86e00b11e0 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e87d5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f86e0066ea0 con 0x7f86e015c0f0 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e87d5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f86e00670f0 con 0x7f86e00b2a20 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e6571640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86e015c0f0 0x7f86e00b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e5d70640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f86e00b11e0 0x7f86e00ad1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44888/0 (socket says 172.21.15.77:44888) 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e5d70640 1 -- 172.21.15.77:0/4174884685 learned_addr learned my addr 172.21.15.77:0/4174884685 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e6d72640 1 --2- 172.21.15.77:0/4174884685 >> v2:172.21.15.121:3300/0 conn(0x7f86e00b2a20 0x7f86e00b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e6571640 1 -- 172.21.15.77:0/4174884685 >> v2:172.21.15.121:3300/0 conn(0x7f86e00b2a20 msgr2=0x7f86e00b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e6571640 1 --2- 172.21.15.77:0/4174884685 >> v2:172.21.15.121:3300/0 conn(0x7f86e00b2a20 0x7f86e00b5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e6571640 1 -- 172.21.15.77:0/4174884685 >> v1:172.21.15.77:6789/0 conn(0x7f86e00b11e0 legacy=0x7f86e00ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e556f640 1 -- 172.21.15.77:0/4174884685 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793461854 0 0) 0x7f86d0001110 con 0x7f86e00b11e0 2024-04-25T01:23:13.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e6571640 1 -- 172.21.15.77:0/4174884685 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e00673d0 con 0x7f86e015c0f0 2024-04-25T01:23:13.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.606+0000 7f86e6d72640 1 --2- 172.21.15.77:0/4174884685 >> v2:172.21.15.121:3300/0 conn(0x7f86e00b2a20 0x7f86e00b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:13.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.607+0000 7f86e6571640 1 --2- 172.21.15.77:0/4174884685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e015c0f0 0x7f86e00b0ca0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f86e0072280 tx=0x7f86dc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c40a76f005ef430 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.607+0000 7f86e556f640 1 -- 172.21.15.77:0/4174884685 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc037070 con 0x7f86e015c0f0 2024-04-25T01:23:13.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.607+0000 7f86e556f640 1 -- 172.21.15.77:0/4174884685 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86dc032440 con 0x7f86e015c0f0 2024-04-25T01:23:13.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.607+0000 7f86e556f640 1 -- 172.21.15.77:0/4174884685 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc032810 con 0x7f86e015c0f0 2024-04-25T01:23:13.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.608+0000 7f86e87d5640 1 -- 172.21.15.77:0/4174884685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e015c0f0 msgr2=0x7f86e00b0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.608+0000 7f86e87d5640 1 --2- 172.21.15.77:0/4174884685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e015c0f0 0x7f86e00b0ca0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f86e0072280 tx=0x7f86dc02a230 comp rx=0 tx=0).stop 2024-04-25T01:23:13.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.608+0000 7f86e87d5640 1 -- 172.21.15.77:0/4174884685 shutdown_connections 2024-04-25T01:23:13.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.608+0000 7f86e87d5640 1 --2- 172.21.15.77:0/4174884685 >> v2:172.21.15.121:3300/0 conn(0x7f86e00b2a20 0x7f86e00b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.608+0000 7f86e87d5640 1 --2- 172.21.15.77:0/4174884685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e015c0f0 0x7f86e00b0ca0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.608+0000 7f86e87d5640 1 -- 172.21.15.77:0/4174884685 >> 172.21.15.77:0/4174884685 conn(0x7f86e00b1f00 msgr2=0x7f86e00b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:13.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.608+0000 7f86e87d5640 1 -- 172.21.15.77:0/4174884685 shutdown_connections 2024-04-25T01:23:13.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.608+0000 7f86e87d5640 1 -- 172.21.15.77:0/4174884685 wait complete. 2024-04-25T01:23:13.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.609+0000 7f86e87d5640 1 Processor -- start 2024-04-25T01:23:13.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.609+0000 7f86e87d5640 1 -- start start 2024-04-25T01:23:13.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e87d5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e00b11e0 0x7f86e01f36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e87d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86e00b2a20 0x7f86e00ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e87d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e015c0f0 0x7f86e0158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e87d5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86e00b00c0 con 0x7f86e015c0f0 2024-04-25T01:23:13.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e87d5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f86e00b0780 con 0x7f86e00b11e0 2024-04-25T01:23:13.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e87d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f86e00b0240 con 0x7f86e00b2a20 2024-04-25T01:23:13.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e6d72640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e015c0f0 0x7f86e0158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e6d72640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e015c0f0 0x7f86e0158330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51398/0 (socket says 172.21.15.77:51398) 2024-04-25T01:23:13.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e6d72640 1 -- 172.21.15.77:0/1007751665 learned_addr learned my addr 172.21.15.77:0/1007751665 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e6571640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e00b11e0 0x7f86e01f36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e5d70640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86e00b2a20 0x7f86e00ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e6d72640 1 -- 172.21.15.77:0/1007751665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86e00b2a20 msgr2=0x7f86e00ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e6d72640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86e00b2a20 0x7f86e00ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e6d72640 1 -- 172.21.15.77:0/1007751665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e00b11e0 msgr2=0x7f86e01f36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e6d72640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e00b11e0 0x7f86e01f36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e6d72640 1 -- 172.21.15.77:0/1007751665 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e01f3c20 con 0x7f86e015c0f0 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e5d70640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86e00b2a20 0x7f86e00ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e6d72640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e015c0f0 0x7f86e0158330 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f86d8007ef0 tx=0x7f86d800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86d6ffd640 1 -- 172.21.15.77:0/1007751665 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d801a070 con 0x7f86e015c0f0 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86d6ffd640 1 -- 172.21.15.77:0/1007751665 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d8003b90 con 0x7f86e015c0f0 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e87d5640 1 -- 172.21.15.77:0/1007751665 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e00cd430 con 0x7f86e015c0f0 2024-04-25T01:23:13.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86d6ffd640 1 -- 172.21.15.77:0/1007751665 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d800de20 con 0x7f86e015c0f0 2024-04-25T01:23:13.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.610+0000 7f86e87d5640 1 -- 172.21.15.77:0/1007751665 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e00c9910 con 0x7f86e015c0f0 2024-04-25T01:23:13.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.611+0000 7f86e87d5640 1 -- 172.21.15.77:0/1007751665 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86e0068fb0 con 0x7f86e015c0f0 2024-04-25T01:23:13.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.612+0000 7f86d6ffd640 1 -- 172.21.15.77:0/1007751665 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f86d801d070 con 0x7f86e015c0f0 2024-04-25T01:23:13.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.612+0000 7f86d6ffd640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86c406be30 0x7f86c406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.612+0000 7f86d6ffd640 1 -- 172.21.15.77:0/1007751665 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f86d808b1c0 con 0x7f86e015c0f0 2024-04-25T01:23:13.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.612+0000 7f86e6571640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86c406be30 0x7f86c406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.613+0000 7f86e6571640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86c406be30 0x7f86c406e2f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f86dc034000 tx=0x7f86dc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.616+0000 7f86d6ffd640 1 -- 172.21.15.77:0/1007751665 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f86d8010070 con 0x7f86e015c0f0 2024-04-25T01:23:13.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.793+0000 7f86e87d5640 1 -- 172.21.15.77:0/1007751665 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f86e0067880 con 0x7f86e015c0f0 2024-04-25T01:23:13.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.793+0000 7f86d6ffd640 1 -- 172.21.15.77:0/1007751665 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7f86d8059340 con 0x7f86e015c0f0 2024-04-25T01:23:13.774 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:23:13.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 -- 172.21.15.77:0/1007751665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86c406be30 msgr2=0x7f86c406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86c406be30 0x7f86c406e2f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f86dc034000 tx=0x7f86dc02e000 comp rx=0 tx=0).stop 2024-04-25T01:23:13.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 -- 172.21.15.77:0/1007751665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e015c0f0 msgr2=0x7f86e0158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e015c0f0 0x7f86e0158330 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f86d8007ef0 tx=0x7f86d800d4a0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 -- 172.21.15.77:0/1007751665 shutdown_connections 2024-04-25T01:23:13.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f86c406be30 0x7f86c406e2f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e015c0f0 0x7f86e0158330 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86e00b2a20 0x7f86e00ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 --2- 172.21.15.77:0/1007751665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e00b11e0 0x7f86e01f36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 -- 172.21.15.77:0/1007751665 >> 172.21.15.77:0/1007751665 conn(0x7f86e00b1f00 msgr2=0x7f86e01599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:13.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 -- 172.21.15.77:0/1007751665 shutdown_connections 2024-04-25T01:23:13.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.795+0000 7f86e87d5640 1 -- 172.21.15.77:0/1007751665 wait complete. 2024-04-25T01:23:13.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:23:13.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.904+0000 7f4e9bc3f640 1 Processor -- start 2024-04-25T01:23:13.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.904+0000 7f4e9bc3f640 1 -- start start 2024-04-25T01:23:13.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e9bc3f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e9415e4c0 0x7f4e940d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e9bc3f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4e940d4a10 0x7f4e940b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e9bc3f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4e940d2d40 0x7f4e940b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e9bc3f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f4e940d1e90 con 0x7f4e940d4a10 2024-04-25T01:23:13.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e9bc3f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4e940b54a0 con 0x7f4e940d2d40 2024-04-25T01:23:13.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e9bc3f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4e940d2130 con 0x7f4e9415e4c0 2024-04-25T01:23:13.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e991da640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4e940d4a10 0x7f4e940b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e991da640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4e940d4a10 0x7f4e940b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51402/0 (socket says 172.21.15.77:51402) 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e991da640 1 -- 172.21.15.77:0/42819015 learned_addr learned my addr 172.21.15.77:0/42819015 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e9a1dc640 1 --2- 172.21.15.77:0/42819015 >> v2:172.21.15.121:3300/0 conn(0x7f4e940d2d40 0x7f4e940b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e999db640 1 --2- 172.21.15.77:0/42819015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e9415e4c0 0x7f4e940d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e991da640 1 -- 172.21.15.77:0/42819015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e9415e4c0 msgr2=0x7f4e940d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e991da640 1 --2- 172.21.15.77:0/42819015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e9415e4c0 0x7f4e940d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e991da640 1 -- 172.21.15.77:0/42819015 >> v2:172.21.15.121:3300/0 conn(0x7f4e940d2d40 msgr2=0x7f4e940b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e991da640 1 --2- 172.21.15.77:0/42819015 >> v2:172.21.15.121:3300/0 conn(0x7f4e940d2d40 0x7f4e940b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e991da640 1 -- 172.21.15.77:0/42819015 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e940b5780 con 0x7f4e940d4a10 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.905+0000 7f4e9a1dc640 1 --2- 172.21.15.77:0/42819015 >> v2:172.21.15.121:3300/0 conn(0x7f4e940d2d40 0x7f4e940b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.906+0000 7f4e991da640 1 --2- 172.21.15.77:0/42819015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e940d4a10 0x7f4e940b3ee0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4e9408e630 tx=0x7f4e8402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b238852a709557d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.906+0000 7f4e989d9640 1 -- 172.21.15.77:0/42819015 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84037070 con 0x7f4e940d4a10 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.906+0000 7f4e989d9640 1 -- 172.21.15.77:0/42819015 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e84032440 con 0x7f4e940d4a10 2024-04-25T01:23:13.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.906+0000 7f4e989d9640 1 -- 172.21.15.77:0/42819015 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84032810 con 0x7f4e940d4a10 2024-04-25T01:23:13.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.906+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/42819015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e940d4a10 msgr2=0x7f4e940b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.906+0000 7f4e9bc3f640 1 --2- 172.21.15.77:0/42819015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e940d4a10 0x7f4e940b3ee0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4e9408e630 tx=0x7f4e8402a140 comp rx=0 tx=0).stop 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.907+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/42819015 shutdown_connections 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.907+0000 7f4e9bc3f640 1 --2- 172.21.15.77:0/42819015 >> v2:172.21.15.121:3300/0 conn(0x7f4e940d2d40 0x7f4e940b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.907+0000 7f4e9bc3f640 1 --2- 172.21.15.77:0/42819015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e940d4a10 0x7f4e940b3ee0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.907+0000 7f4e9bc3f640 1 --2- 172.21.15.77:0/42819015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e9415e4c0 0x7f4e940d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.907+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/42819015 >> 172.21.15.77:0/42819015 conn(0x7f4e940d3ef0 msgr2=0x7f4e940d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.907+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/42819015 shutdown_connections 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.907+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/42819015 wait complete. 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.908+0000 7f4e9bc3f640 1 Processor -- start 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.908+0000 7f4e9bc3f640 1 -- start start 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.908+0000 7f4e9bc3f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e940d2d40 0x7f4e940cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9bc3f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e940d4a10 0x7f4e940ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9bc3f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e9415e4c0 0x7f4e9415c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9bc3f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9415ea20 con 0x7f4e9415e4c0 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9bc3f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9415e8a0 con 0x7f4e940d4a10 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9bc3f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4e940d0900 con 0x7f4e940d2d40 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9a1dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e9415e4c0 0x7f4e9415c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9a1dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e9415e4c0 0x7f4e9415c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51414/0 (socket says 172.21.15.77:51414) 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9a1dc640 1 -- 172.21.15.77:0/4014814402 learned_addr learned my addr 172.21.15.77:0/4014814402 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e991da640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e940d4a10 0x7f4e940ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e999db640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e940d2d40 0x7f4e940cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9a1dc640 1 -- 172.21.15.77:0/4014814402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e940d2d40 msgr2=0x7f4e940cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9a1dc640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e940d2d40 0x7f4e940cdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9a1dc640 1 -- 172.21.15.77:0/4014814402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e940d4a10 msgr2=0x7f4e940ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9a1dc640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e940d4a10 0x7f4e940ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9a1dc640 1 -- 172.21.15.77:0/4014814402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9415b190 con 0x7f4e9415e4c0 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e991da640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e940d4a10 0x7f4e940ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e999db640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e940d2d40 0x7f4e940cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9a1dc640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e9415e4c0 0x7f4e9415c230 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c007e40 tx=0x7f4e8c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e8a7fc640 1 -- 172.21.15.77:0/4014814402 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c01a070 con 0x7f4e9415e4c0 2024-04-25T01:23:13.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e8a7fc640 1 -- 172.21.15.77:0/4014814402 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8c003b90 con 0x7f4e9415e4c0 2024-04-25T01:23:13.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/4014814402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e9415b470 con 0x7f4e9415e4c0 2024-04-25T01:23:13.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e8a7fc640 1 -- 172.21.15.77:0/4014814402 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c00de80 con 0x7f4e9415e4c0 2024-04-25T01:23:13.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.909+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/4014814402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e9415ba30 con 0x7f4e9415e4c0 2024-04-25T01:23:13.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.910+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/4014814402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e9406d240 con 0x7f4e9415e4c0 2024-04-25T01:23:13.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.911+0000 7f4e8a7fc640 1 -- 172.21.15.77:0/4014814402 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4e8c01d070 con 0x7f4e9415e4c0 2024-04-25T01:23:13.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.911+0000 7f4e8a7fc640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e7806be30 0x7f4e7806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.911+0000 7f4e8a7fc640 1 -- 172.21.15.77:0/4014814402 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f4e8c08b260 con 0x7f4e9415e4c0 2024-04-25T01:23:13.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.911+0000 7f4e999db640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e7806be30 0x7f4e7806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.911+0000 7f4e999db640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e7806be30 0x7f4e7806e2f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4e90006fd0 tx=0x7f4e90008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:13.915+0000 7f4e8a7fc640 1 -- 172.21.15.77:0/4014814402 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4e9406d240 con 0x7f4e9415e4c0 2024-04-25T01:23:14.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.097+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/4014814402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4e94068fb0 con 0x7f4e9415e4c0 2024-04-25T01:23:14.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.098+0000 7f4e8a7fc640 1 -- 172.21.15.77:0/4014814402 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v39) ==== 80+0+122 (secure 0 0 0) 0x7f4e8c010070 con 0x7f4e9415e4c0 2024-04-25T01:23:14.078 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6837/2944320661 2024-04-26T01:23:06.238820+0000 2024-04-25T01:23:14.078 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6836/2944320661 2024-04-26T01:23:06.238820+0000 2024-04-25T01:23:14.079 INFO:teuthology.orchestra.run.smithi077.stderr:listed 2 entries 2024-04-25T01:23:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/4014814402 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e7806be30 msgr2=0x7f4e7806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e7806be30 0x7f4e7806e2f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4e90006fd0 tx=0x7f4e90008040 comp rx=0 tx=0).stop 2024-04-25T01:23:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/4014814402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e9415e4c0 msgr2=0x7f4e9415c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e9415e4c0 0x7f4e9415c230 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c007e40 tx=0x7f4e8c00d4a0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/4014814402 shutdown_connections 2024-04-25T01:23:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f4e7806be30 0x7f4e7806e2f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e9415e4c0 0x7f4e9415c230 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e940d4a10 0x7f4e940ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 --2- 172.21.15.77:0/4014814402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e940d2d40 0x7f4e940cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/4014814402 >> 172.21.15.77:0/4014814402 conn(0x7f4e940d3ef0 msgr2=0x7f4e940747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:14.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/4014814402 shutdown_connections 2024-04-25T01:23:14.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.100+0000 7f4e9bc3f640 1 -- 172.21.15.77:0/4014814402 wait complete. 2024-04-25T01:23:14.090 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T01:23:14.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:14.091 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:14.117 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:23:14.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:14.117 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T01:23:14.172 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:23:14.172 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:23:14.172 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:14.191 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=434 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:23:14.191 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:23:14.191 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:23:14.191 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:14.227 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T01:23:14.227 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:23:14.227 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:23:14.227 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:14.248 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=434 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:23:14.248 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:14.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.383+0000 7fafc46ee640 1 Processor -- start 2024-04-25T01:23:14.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.383+0000 7fafc46ee640 1 -- start start 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.384+0000 7fafc46ee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafbc15e4c0 0x7fafbc0d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.384+0000 7fafc46ee640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fafbc0d2d40 0x7fafbc0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.384+0000 7fafc46ee640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafbc0d1e90 con 0x7fafbc0d4a10 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.384+0000 7fafc46ee640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fafbc0b54a0 con 0x7fafbc0d2d40 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.384+0000 7fafc46ee640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fafbc0d2130 con 0x7fafbc15e4c0 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.384+0000 7fafc2c8b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fafbc0d2d40 0x7fafbc0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.384+0000 7fafc1c89640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fafbc0d4a10 0x7fafbc0b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44898/0 (socket says 172.21.15.77:44898) 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.384+0000 7fafc1c89640 1 -- 172.21.15.77:0/1036058455 learned_addr learned my addr 172.21.15.77:0/1036058455 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.385+0000 7fafc2c8b640 1 -- 172.21.15.77:0/1036058455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafbc15e4c0 msgr2=0x7fafbc0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.385+0000 7fafc2c8b640 1 --2- 172.21.15.77:0/1036058455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafbc15e4c0 0x7fafbc0d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.385+0000 7fafc2c8b640 1 -- 172.21.15.77:0/1036058455 >> v1:172.21.15.77:6789/0 conn(0x7fafbc0d4a10 legacy=0x7fafbc0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.385+0000 7fafc2c8b640 1 -- 172.21.15.77:0/1036058455 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafbc0b5780 con 0x7fafbc0d2d40 2024-04-25T01:23:14.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.385+0000 7fafc1488640 1 -- 172.21.15.77:0/1036058455 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2646363226 0 0) 0x7fafbc0d1e90 con 0x7fafbc0d4a10 2024-04-25T01:23:14.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.385+0000 7fafc2c8b640 1 --2- 172.21.15.77:0/1036058455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafbc0d2d40 0x7fafbc0cfaf0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fafb40097a0 tx=0x7fafb402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e83ecced46cf72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:14.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.385+0000 7fafc1488640 1 -- 172.21.15.77:0/1036058455 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb4037070 con 0x7fafbc0d2d40 2024-04-25T01:23:14.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.385+0000 7fafc1488640 1 -- 172.21.15.77:0/1036058455 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafb402ae50 con 0x7fafbc0d2d40 2024-04-25T01:23:14.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.385+0000 7fafc1488640 1 -- 172.21.15.77:0/1036058455 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb4032660 con 0x7fafbc0d2d40 2024-04-25T01:23:14.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.386+0000 7fafc46ee640 1 -- 172.21.15.77:0/1036058455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafbc0d2d40 msgr2=0x7fafbc0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.386+0000 7fafc46ee640 1 --2- 172.21.15.77:0/1036058455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafbc0d2d40 0x7fafbc0cfaf0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fafb40097a0 tx=0x7fafb402a100 comp rx=0 tx=0).stop 2024-04-25T01:23:14.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.386+0000 7fafc46ee640 1 -- 172.21.15.77:0/1036058455 shutdown_connections 2024-04-25T01:23:14.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.386+0000 7fafc46ee640 1 --2- 172.21.15.77:0/1036058455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafbc0d2d40 0x7fafbc0cfaf0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.386+0000 7fafc46ee640 1 --2- 172.21.15.77:0/1036058455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafbc15e4c0 0x7fafbc0d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.386+0000 7fafc46ee640 1 -- 172.21.15.77:0/1036058455 >> 172.21.15.77:0/1036058455 conn(0x7fafbc0d3ef0 msgr2=0x7fafbc0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:14.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.386+0000 7fafc46ee640 1 -- 172.21.15.77:0/1036058455 shutdown_connections 2024-04-25T01:23:14.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.386+0000 7fafc46ee640 1 -- 172.21.15.77:0/1036058455 wait complete. 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.387+0000 7fafc46ee640 1 Processor -- start 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.387+0000 7fafc46ee640 1 -- start start 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc46ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafbc0d2d40 0x7fafbc0cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc46ee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafbc0d4a10 0x7fafbc14e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc46ee640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafbc15e4c0 0x7fafbc15ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc46ee640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fafbc15ea20 con 0x7fafbc0d2d40 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc46ee640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fafbc15e8a0 con 0x7fafbc15e4c0 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc46ee640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fafbc0b45f0 con 0x7fafbc0d4a10 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc248a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafbc0d2d40 0x7fafbc0cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc248a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafbc0d2d40 0x7fafbc0cafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51438/0 (socket says 172.21.15.77:51438) 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc248a640 1 -- 172.21.15.77:0/3036383143 learned_addr learned my addr 172.21.15.77:0/3036383143 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc1c89640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafbc0d4a10 0x7fafbc14e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc2c8b640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafbc15e4c0 0x7fafbc15ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc248a640 1 -- 172.21.15.77:0/3036383143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafbc0d4a10 msgr2=0x7fafbc14e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc248a640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafbc0d4a10 0x7fafbc14e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc248a640 1 -- 172.21.15.77:0/3036383143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafbc15e4c0 msgr2=0x7fafbc15ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc248a640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafbc15e4c0 0x7fafbc15ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc248a640 1 -- 172.21.15.77:0/3036383143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafbc14e870 con 0x7fafbc0d2d40 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc1c89640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafbc0d4a10 0x7fafbc14e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc2c8b640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafbc15e4c0 0x7fafbc15ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.388+0000 7fafc248a640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafbc0d2d40 0x7fafbc0cafc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fafb800ba50 tx=0x7fafb800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.389+0000 7fafb2ffd640 1 -- 172.21.15.77:0/3036383143 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8019070 con 0x7fafbc0d2d40 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.389+0000 7fafb2ffd640 1 -- 172.21.15.77:0/3036383143 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafb800cc00 con 0x7fafbc0d2d40 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.389+0000 7fafc46ee640 1 -- 172.21.15.77:0/3036383143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafbc0cb5a0 con 0x7fafbc0d2d40 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.389+0000 7fafb2ffd640 1 -- 172.21.15.77:0/3036383143 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8014440 con 0x7fafbc0d2d40 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.389+0000 7fafc46ee640 1 -- 172.21.15.77:0/3036383143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafbc0cbb60 con 0x7fafbc0d2d40 2024-04-25T01:23:14.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.390+0000 7fafc46ee640 1 -- 172.21.15.77:0/3036383143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafbc068fb0 con 0x7fafbc0d2d40 2024-04-25T01:23:14.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.390+0000 7fafb2ffd640 1 -- 172.21.15.77:0/3036383143 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fafb801c070 con 0x7fafbc0d2d40 2024-04-25T01:23:14.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.390+0000 7fafb2ffd640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fafa006be30 0x7fafa006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.391+0000 7fafb2ffd640 1 -- 172.21.15.77:0/3036383143 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7fafb808ae10 con 0x7fafbc0d2d40 2024-04-25T01:23:14.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.391+0000 7fafc1c89640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fafa006be30 0x7fafa006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.391+0000 7fafc1c89640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fafa006be30 0x7fafa006e2f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fafbc151ce0 tx=0x7fafac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:14.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.395+0000 7fafb2ffd640 1 -- 172.21.15.77:0/3036383143 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fafbc14e870 con 0x7fafbc0d2d40 2024-04-25T01:23:14.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.603+0000 7fafc46ee640 1 -- 172.21.15.77:0/3036383143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafbc0d3120 con 0x7fafbc0d2d40 2024-04-25T01:23:14.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.603+0000 7fafb2ffd640 1 -- 172.21.15.77:0/3036383143 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5188 (secure 0 0 0) 0x7fafb80595e0 con 0x7fafbc0d2d40 2024-04-25T01:23:14.584 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:14.584 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2823778882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2823778882},{"type":"v1","addr":"172.21.15.77:6837","nonce":2823778882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.066906+0000","modified":"2024-04-25T01:23:09.251757+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5355":{"gid":5355,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6835/3139470828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3139470828},{"type":"v1","addr":"172.21.15.77:6835","nonce":3139470828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5355,"qdb_cluster":[5355]},"id":2}]} 2024-04-25T01:23:14.584 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-04-25T01:23:14.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 -- 172.21.15.77:0/3036383143 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fafa006be30 msgr2=0x7fafa006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fafa006be30 0x7fafa006e2f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fafbc151ce0 tx=0x7fafac009040 comp rx=0 tx=0).stop 2024-04-25T01:23:14.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 -- 172.21.15.77:0/3036383143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafbc0d2d40 msgr2=0x7fafbc0cafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafbc0d2d40 0x7fafbc0cafc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fafb800ba50 tx=0x7fafb800bf10 comp rx=0 tx=0).stop 2024-04-25T01:23:14.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 -- 172.21.15.77:0/3036383143 shutdown_connections 2024-04-25T01:23:14.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fafa006be30 0x7fafa006e2f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafbc15e4c0 0x7fafbc15ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafbc0d4a10 0x7fafbc14e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 --2- 172.21.15.77:0/3036383143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafbc0d2d40 0x7fafbc0cafc0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 -- 172.21.15.77:0/3036383143 >> 172.21.15.77:0/3036383143 conn(0x7fafbc0d3ef0 msgr2=0x7fafbc06c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:14.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 -- 172.21.15.77:0/3036383143 shutdown_connections 2024-04-25T01:23:14.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.605+0000 7fafc46ee640 1 -- 172.21.15.77:0/3036383143 wait complete. 2024-04-25T01:23:14.597 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:14.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.712+0000 7f5e3e872640 1 Processor -- start 2024-04-25T01:23:14.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.712+0000 7f5e3e872640 1 -- start start 2024-04-25T01:23:14.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.713+0000 7f5e3e872640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5e3815e4c0 0x7f5e380d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.713+0000 7f5e3e872640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 0x7f5e380cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.713+0000 7f5e3e872640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e380d1e90 con 0x7f5e380d4a10 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.713+0000 7f5e3e872640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f5e380b54a0 con 0x7f5e3815e4c0 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.713+0000 7f5e3e872640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5e380d2130 con 0x7f5e380d2d40 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.713+0000 7f5e37fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5e3815e4c0 0x7f5e380d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.713+0000 7f5e377fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5e380d4a10 0x7f5e380b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44914/0 (socket says 172.21.15.77:44914) 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.713+0000 7f5e377fe640 1 -- 172.21.15.77:0/1320547739 learned_addr learned my addr 172.21.15.77:0/1320547739 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.714+0000 7f5e3ce0f640 1 --2- 172.21.15.77:0/1320547739 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 0x7f5e380cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.714+0000 7f5e37fff640 1 -- 172.21.15.77:0/1320547739 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 msgr2=0x7f5e380cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.714+0000 7f5e37fff640 1 --2- 172.21.15.77:0/1320547739 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 0x7f5e380cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.714+0000 7f5e37fff640 1 -- 172.21.15.77:0/1320547739 >> v1:172.21.15.77:6789/0 conn(0x7f5e380d4a10 legacy=0x7f5e380b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.714+0000 7f5e36ffd640 1 -- 172.21.15.77:0/1320547739 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2028790393 0 0) 0x7f5e380d1e90 con 0x7f5e380d4a10 2024-04-25T01:23:14.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.714+0000 7f5e37fff640 1 -- 172.21.15.77:0/1320547739 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e380b5780 con 0x7f5e3815e4c0 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.714+0000 7f5e3ce0f640 1 --2- 172.21.15.77:0/1320547739 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 0x7f5e380cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.714+0000 7f5e37fff640 1 --2- 172.21.15.77:0/1320547739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e3815e4c0 0x7f5e380d2800 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5e280097a0 tx=0x7f5e2802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36df48e11ed6ef9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.715+0000 7f5e36ffd640 1 -- 172.21.15.77:0/1320547739 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e28037070 con 0x7f5e3815e4c0 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.715+0000 7f5e36ffd640 1 -- 172.21.15.77:0/1320547739 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e2802ae50 con 0x7f5e3815e4c0 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.715+0000 7f5e36ffd640 1 -- 172.21.15.77:0/1320547739 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e28032660 con 0x7f5e3815e4c0 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.715+0000 7f5e3e872640 1 -- 172.21.15.77:0/1320547739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e3815e4c0 msgr2=0x7f5e380d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.715+0000 7f5e3e872640 1 --2- 172.21.15.77:0/1320547739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e3815e4c0 0x7f5e380d2800 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5e280097a0 tx=0x7f5e2802a100 comp rx=0 tx=0).stop 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.715+0000 7f5e3e872640 1 -- 172.21.15.77:0/1320547739 shutdown_connections 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.715+0000 7f5e3e872640 1 --2- 172.21.15.77:0/1320547739 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 0x7f5e380cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.715+0000 7f5e3e872640 1 --2- 172.21.15.77:0/1320547739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e3815e4c0 0x7f5e380d2800 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.715+0000 7f5e3e872640 1 -- 172.21.15.77:0/1320547739 >> 172.21.15.77:0/1320547739 conn(0x7f5e380d3ef0 msgr2=0x7f5e380d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.715+0000 7f5e3e872640 1 -- 172.21.15.77:0/1320547739 shutdown_connections 2024-04-25T01:23:14.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.715+0000 7f5e3e872640 1 -- 172.21.15.77:0/1320547739 wait complete. 2024-04-25T01:23:14.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.716+0000 7f5e3e872640 1 Processor -- start 2024-04-25T01:23:14.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.717+0000 7f5e3e872640 1 -- start start 2024-04-25T01:23:14.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.717+0000 7f5e3e872640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 0x7f5e3814b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.717+0000 7f5e3e872640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e380d4a10 0x7f5e380b8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.717+0000 7f5e3e872640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e3815e4c0 0x7f5e380b9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.717+0000 7f5e3e872640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3815ea20 con 0x7f5e380d4a10 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.717+0000 7f5e3e872640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3815e8a0 con 0x7f5e3815e4c0 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.717+0000 7f5e3e872640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5e380b45f0 con 0x7f5e380d2d40 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.717+0000 7f5e377fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e380d4a10 0x7f5e380b8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.717+0000 7f5e377fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e380d4a10 0x7f5e380b8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51448/0 (socket says 172.21.15.77:51448) 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.717+0000 7f5e377fe640 1 -- 172.21.15.77:0/1688797712 learned_addr learned my addr 172.21.15.77:0/1688797712 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e37fff640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 0x7f5e3814b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e3ce0f640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e3815e4c0 0x7f5e380b9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e377fe640 1 -- 172.21.15.77:0/1688797712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 msgr2=0x7f5e3814b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e377fe640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 0x7f5e3814b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e377fe640 1 -- 172.21.15.77:0/1688797712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e3815e4c0 msgr2=0x7f5e380b9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e377fe640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e3815e4c0 0x7f5e380b9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e377fe640 1 -- 172.21.15.77:0/1688797712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e380b9bb0 con 0x7f5e380d4a10 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e37fff640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 0x7f5e3814b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e377fe640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e380d4a10 0x7f5e380b8f30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00c4b0 tx=0x7f5e2c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e34ff9640 1 -- 172.21.15.77:0/1688797712 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c018070 con 0x7f5e380d4a10 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e34ff9640 1 -- 172.21.15.77:0/1688797712 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e2c007500 con 0x7f5e380d4a10 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e3e872640 1 -- 172.21.15.77:0/1688797712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e380c8c10 con 0x7f5e380d4a10 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e34ff9640 1 -- 172.21.15.77:0/1688797712 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c004a20 con 0x7f5e380d4a10 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.718+0000 7f5e3e872640 1 -- 172.21.15.77:0/1688797712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e380c96f0 con 0x7f5e380d4a10 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.719+0000 7f5e3e872640 1 -- 172.21.15.77:0/1688797712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e38068fb0 con 0x7f5e380d4a10 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.719+0000 7f5e34ff9640 1 -- 172.21.15.77:0/1688797712 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5e2c01b070 con 0x7f5e380d4a10 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.720+0000 7f5e34ff9640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e2406be60 0x7f5e2406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.720+0000 7f5e34ff9640 1 -- 172.21.15.77:0/1688797712 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f5e2c08a2e0 con 0x7f5e380d4a10 2024-04-25T01:23:14.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.720+0000 7f5e37fff640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e2406be60 0x7f5e2406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.720+0000 7f5e37fff640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e2406be60 0x7f5e2406e320 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5e28034000 tx=0x7f5e2802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:14.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.724+0000 7f5e34ff9640 1 -- 172.21.15.77:0/1688797712 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5e2c058c00 con 0x7f5e380d4a10 2024-04-25T01:23:14.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.894+0000 7f5e3e872640 1 -- 172.21.15.77:0/1688797712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e38159960 con 0x7f5e380d4a10 2024-04-25T01:23:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.896+0000 7f5e34ff9640 1 -- 172.21.15.77:0/1688797712 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+17071 (secure 0 0 0) 0x7f5e2c058500 con 0x7f5e380d4a10 2024-04-25T01:23:14.876 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:14.876 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":39,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:23:06.238842+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.457887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.029883+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6837/2944320661":"2024-04-26T01:23:06.238820+0000","172.21.15.77:6836/2944320661":"2024-04-26T01:23:06.238820+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.897+0000 7f5e3e872640 1 -- 172.21.15.77:0/1688797712 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e2406be60 msgr2=0x7f5e2406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.897+0000 7f5e3e872640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e2406be60 0x7f5e2406e320 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5e28034000 tx=0x7f5e2802e000 comp rx=0 tx=0).stop 2024-04-25T01:23:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.897+0000 7f5e3e872640 1 -- 172.21.15.77:0/1688797712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e380d4a10 msgr2=0x7f5e380b8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.897+0000 7f5e3e872640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e380d4a10 0x7f5e380b8f30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00c4b0 tx=0x7f5e2c00c970 comp rx=0 tx=0).stop 2024-04-25T01:23:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.898+0000 7f5e3e872640 1 -- 172.21.15.77:0/1688797712 shutdown_connections 2024-04-25T01:23:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.898+0000 7f5e3e872640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e2406be60 0x7f5e2406e320 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.898+0000 7f5e3e872640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e380d4a10 0x7f5e380b8f30 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.898+0000 7f5e3e872640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e3815e4c0 0x7f5e380b9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.898+0000 7f5e3e872640 1 --2- 172.21.15.77:0/1688797712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e380d2d40 0x7f5e3814b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.898+0000 7f5e3e872640 1 -- 172.21.15.77:0/1688797712 >> 172.21.15.77:0/1688797712 conn(0x7f5e380d3ef0 msgr2=0x7f5e380ceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.898+0000 7f5e3e872640 1 -- 172.21.15.77:0/1688797712 shutdown_connections 2024-04-25T01:23:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:14.898+0000 7f5e3e872640 1 -- 172.21.15.77:0/1688797712 wait complete. 2024-04-25T01:23:14.889 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.457887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.029883+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:23:14.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:23:14.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.007+0000 7f25f8fc1640 1 Processor -- start 2024-04-25T01:23:14.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.007+0000 7f25f8fc1640 1 -- start start 2024-04-25T01:23:14.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.008+0000 7f25f8fc1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25f40d23d0 0x7f25f40d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.008+0000 7f25f8fc1640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f25f40d27b0 0x7f25f40d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.008+0000 7f25f8fc1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f25f40b9380 0x7f25f40b97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.008+0000 7f25f8fc1640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f25f40b9d30 con 0x7f25f40b9380 2024-04-25T01:23:14.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.008+0000 7f25f8fc1640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f25f40cea40 con 0x7f25f40d27b0 2024-04-25T01:23:14.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.008+0000 7f25f8fc1640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f25f40cece0 con 0x7f25f40d23d0 2024-04-25T01:23:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.008+0000 7f25f2d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f25f40b9380 0x7f25f40b97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.008+0000 7f25f2d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f25f40b9380 0x7f25f40b97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51454/0 (socket says 172.21.15.77:51454) 2024-04-25T01:23:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.008+0000 7f25f2d9d640 1 -- 172.21.15.77:0/3194315334 learned_addr learned my addr 172.21.15.77:0/3194315334 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.009+0000 7f25f259c640 1 --2- 172.21.15.77:0/3194315334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25f40d23d0 0x7f25f40d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.009+0000 7f25f1d9b640 1 --2- 172.21.15.77:0/3194315334 >> v2:172.21.15.121:3300/0 conn(0x7f25f40d27b0 0x7f25f40d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.009+0000 7f25f2d9d640 1 -- 172.21.15.77:0/3194315334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25f40d23d0 msgr2=0x7f25f40d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.009+0000 7f25f2d9d640 1 --2- 172.21.15.77:0/3194315334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25f40d23d0 0x7f25f40d4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.009+0000 7f25f2d9d640 1 -- 172.21.15.77:0/3194315334 >> v2:172.21.15.121:3300/0 conn(0x7f25f40d27b0 msgr2=0x7f25f40d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.009+0000 7f25f2d9d640 1 --2- 172.21.15.77:0/3194315334 >> v2:172.21.15.121:3300/0 conn(0x7f25f40d27b0 0x7f25f40d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.009+0000 7f25f2d9d640 1 -- 172.21.15.77:0/3194315334 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f40cefc0 con 0x7f25f40b9380 2024-04-25T01:23:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.009+0000 7f25f1d9b640 1 --2- 172.21.15.77:0/3194315334 >> v2:172.21.15.121:3300/0 conn(0x7f25f40d27b0 0x7f25f40d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.009+0000 7f25f2d9d640 1 --2- 172.21.15.77:0/3194315334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25f40b9380 0x7f25f40b97f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f25f408c7d0 tx=0x7f25ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862e672588345685 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f159a640 1 -- 172.21.15.77:0/3194315334 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec037070 con 0x7f25f40b9380 2024-04-25T01:23:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f159a640 1 -- 172.21.15.77:0/3194315334 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25ec032440 con 0x7f25f40b9380 2024-04-25T01:23:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f159a640 1 -- 172.21.15.77:0/3194315334 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec032810 con 0x7f25f40b9380 2024-04-25T01:23:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f8fc1640 1 -- 172.21.15.77:0/3194315334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25f40b9380 msgr2=0x7f25f40b97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f8fc1640 1 --2- 172.21.15.77:0/3194315334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25f40b9380 0x7f25f40b97f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f25f408c7d0 tx=0x7f25ec02a270 comp rx=0 tx=0).stop 2024-04-25T01:23:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f8fc1640 1 -- 172.21.15.77:0/3194315334 shutdown_connections 2024-04-25T01:23:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f8fc1640 1 --2- 172.21.15.77:0/3194315334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25f40b9380 0x7f25f40b97f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f8fc1640 1 --2- 172.21.15.77:0/3194315334 >> v2:172.21.15.121:3300/0 conn(0x7f25f40d27b0 0x7f25f40d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f8fc1640 1 --2- 172.21.15.77:0/3194315334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25f40d23d0 0x7f25f40d4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f8fc1640 1 -- 172.21.15.77:0/3194315334 >> 172.21.15.77:0/3194315334 conn(0x7f25f40d3e30 msgr2=0x7f25f40d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:14.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f8fc1640 1 -- 172.21.15.77:0/3194315334 shutdown_connections 2024-04-25T01:23:14.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.010+0000 7f25f8fc1640 1 -- 172.21.15.77:0/3194315334 wait complete. 2024-04-25T01:23:14.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.011+0000 7f25f8fc1640 1 Processor -- start 2024-04-25T01:23:14.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f8fc1640 1 -- start start 2024-04-25T01:23:14.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f8fc1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25f40b9380 0x7f25f41f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f8fc1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25f40d23d0 0x7f25f40c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f8fc1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25f40d27b0 0x7f25f414e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f8fc1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25f40cf860 con 0x7f25f40b9380 2024-04-25T01:23:14.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f8fc1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f25f40cf580 con 0x7f25f40d27b0 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f8fc1640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f25f40cfd80 con 0x7f25f40d23d0 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25f40b9380 0x7f25f41f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25f40b9380 0x7f25f41f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51462/0 (socket says 172.21.15.77:51462) 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f259c640 1 -- 172.21.15.77:0/1874300557 learned_addr learned my addr 172.21.15.77:0/1874300557 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f1d9b640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25f40d23d0 0x7f25f40c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.012+0000 7f25f2d9d640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25f40d27b0 0x7f25f414e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.013+0000 7f25f259c640 1 -- 172.21.15.77:0/1874300557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25f40d23d0 msgr2=0x7f25f40c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.013+0000 7f25f259c640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25f40d23d0 0x7f25f40c8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.013+0000 7f25f259c640 1 -- 172.21.15.77:0/1874300557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25f40d27b0 msgr2=0x7f25f414e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.013+0000 7f25f259c640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25f40d27b0 0x7f25f414e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.013+0000 7f25f259c640 1 -- 172.21.15.77:0/1874300557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f4158080 con 0x7f25f40b9380 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.013+0000 7f25f259c640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25f40b9380 0x7f25f41f3d00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f25e400ef00 tx=0x7f25e400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.013+0000 7f25e2ffd640 1 -- 172.21.15.77:0/1874300557 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e4018020 con 0x7f25f40b9380 2024-04-25T01:23:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.013+0000 7f25e2ffd640 1 -- 172.21.15.77:0/1874300557 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25e4014400 con 0x7f25f40b9380 2024-04-25T01:23:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.013+0000 7f25f8fc1640 1 -- 172.21.15.77:0/1874300557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f4158360 con 0x7f25f40b9380 2024-04-25T01:23:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.013+0000 7f25e2ffd640 1 -- 172.21.15.77:0/1874300557 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e40147d0 con 0x7f25f40b9380 2024-04-25T01:23:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.013+0000 7f25f8fc1640 1 -- 172.21.15.77:0/1874300557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f4158920 con 0x7f25f40b9380 2024-04-25T01:23:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.014+0000 7f25f8fc1640 1 -- 172.21.15.77:0/1874300557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25f4068fb0 con 0x7f25f40b9380 2024-04-25T01:23:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.014+0000 7f25e2ffd640 1 -- 172.21.15.77:0/1874300557 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f25e401b020 con 0x7f25f40b9380 2024-04-25T01:23:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.015+0000 7f25e2ffd640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f25d406be30 0x7f25d406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.015+0000 7f25e2ffd640 1 -- 172.21.15.77:0/1874300557 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f25e408b610 con 0x7f25f40b9380 2024-04-25T01:23:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.015+0000 7f25f1d9b640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f25d406be30 0x7f25d406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.015+0000 7f25f1d9b640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f25d406be30 0x7f25d406e2f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f25f40c93d0 tx=0x7f25e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:14.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.019+0000 7f25e2ffd640 1 -- 172.21.15.77:0/1874300557 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f25e4022030 con 0x7f25f40b9380 2024-04-25T01:23:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.192+0000 7f25f8fc1640 1 -- 172.21.15.77:0/1874300557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f25f408df50 con 0x7f25f40b9380 2024-04-25T01:23:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.192+0000 7f25e2ffd640 1 -- 172.21.15.77:0/1874300557 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f25e4059820 con 0x7f25f40b9380 2024-04-25T01:23:15.172 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:23:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 -- 172.21.15.77:0/1874300557 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f25d406be30 msgr2=0x7f25d406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f25d406be30 0x7f25d406e2f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f25f40c93d0 tx=0x7f25e8008040 comp rx=0 tx=0).stop 2024-04-25T01:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 -- 172.21.15.77:0/1874300557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25f40b9380 msgr2=0x7f25f41f3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25f40b9380 0x7f25f41f3d00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f25e400ef00 tx=0x7f25e400b480 comp rx=0 tx=0).stop 2024-04-25T01:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 -- 172.21.15.77:0/1874300557 shutdown_connections 2024-04-25T01:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f25d406be30 0x7f25d406e2f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25f40d27b0 0x7f25f414e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25f40d23d0 0x7f25f40c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 --2- 172.21.15.77:0/1874300557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25f40b9380 0x7f25f41f3d00 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 -- 172.21.15.77:0/1874300557 >> 172.21.15.77:0/1874300557 conn(0x7f25f40d3e30 msgr2=0x7f25f40b6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 -- 172.21.15.77:0/1874300557 shutdown_connections 2024-04-25T01:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.194+0000 7f25f8fc1640 1 -- 172.21.15.77:0/1874300557 wait complete. 2024-04-25T01:23:15.185 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:23:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.305+0000 7f38ca022640 1 Processor -- start 2024-04-25T01:23:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.306+0000 7f38ca022640 1 -- start start 2024-04-25T01:23:15.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.306+0000 7f38ca022640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38c40d23d0 0x7f38c40d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.306+0000 7f38ca022640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38c40cea40 con 0x7f38c40b9a90 2024-04-25T01:23:15.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.306+0000 7f38ca022640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38c40cece0 con 0x7f38c40d27b0 2024-04-25T01:23:15.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.306+0000 7f38ca022640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f38c40cefc0 con 0x7f38c40d23d0 2024-04-25T01:23:15.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.306+0000 7f38c37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38c40d23d0 0x7f38c40d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.306+0000 7f38c3fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f38c40b9a90 0x7f38c40b6c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44926/0 (socket says 172.21.15.77:44926) 2024-04-25T01:23:15.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.306+0000 7f38c3fff640 1 -- 172.21.15.77:0/3154183041 learned_addr learned my addr 172.21.15.77:0/3154183041 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:15.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.306+0000 7f38c37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38c40d23d0 0x7f38c40d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51466/0 (socket says 172.21.15.77:51466) 2024-04-25T01:23:15.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.307+0000 7f38c37fe640 1 -- 172.21.15.77:0/3154183041 >> v1:172.21.15.121:6789/0 conn(0x7f38c40d27b0 legacy=0x7f38c40b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:15.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.307+0000 7f38c37fe640 1 -- 172.21.15.77:0/3154183041 >> v1:172.21.15.77:6789/0 conn(0x7f38c40b9a90 legacy=0x7f38c40b6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.307+0000 7f38c37fe640 1 -- 172.21.15.77:0/3154183041 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c40cf2a0 con 0x7f38c40d23d0 2024-04-25T01:23:15.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.307+0000 7f38c27fc640 1 -- 172.21.15.77:0/3154183041 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 315457977 0 0) 0x7f38c40cea40 con 0x7f38c40b9a90 2024-04-25T01:23:15.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.307+0000 7f38c37fe640 1 --2- 172.21.15.77:0/3154183041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38c40d23d0 0x7f38c40d4920 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f38b40097a0 tx=0x7f38b402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef99a95ce432b70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:15.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.308+0000 7f38c27fc640 1 -- 172.21.15.77:0/3154183041 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b4037070 con 0x7f38c40d23d0 2024-04-25T01:23:15.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.308+0000 7f38c27fc640 1 -- 172.21.15.77:0/3154183041 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b4032640 con 0x7f38c40d23d0 2024-04-25T01:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.308+0000 7f38c27fc640 1 -- 172.21.15.77:0/3154183041 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b4032a10 con 0x7f38c40d23d0 2024-04-25T01:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.308+0000 7f38ca022640 1 -- 172.21.15.77:0/3154183041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38c40d23d0 msgr2=0x7f38c40d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.308+0000 7f38ca022640 1 --2- 172.21.15.77:0/3154183041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38c40d23d0 0x7f38c40d4920 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f38b40097a0 tx=0x7f38b402a2c0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.308+0000 7f38ca022640 1 -- 172.21.15.77:0/3154183041 shutdown_connections 2024-04-25T01:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.308+0000 7f38ca022640 1 --2- 172.21.15.77:0/3154183041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38c40d23d0 0x7f38c40d4920 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.308+0000 7f38ca022640 1 -- 172.21.15.77:0/3154183041 >> 172.21.15.77:0/3154183041 conn(0x7f38c40d3e30 msgr2=0x7f38c40d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.308+0000 7f38ca022640 1 -- 172.21.15.77:0/3154183041 shutdown_connections 2024-04-25T01:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.308+0000 7f38ca022640 1 -- 172.21.15.77:0/3154183041 wait complete. 2024-04-25T01:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.309+0000 7f38ca022640 1 Processor -- start 2024-04-25T01:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.309+0000 7f38ca022640 1 -- start start 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.310+0000 7f38ca022640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38c40b9a90 0x7f38c41f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.310+0000 7f38ca022640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38c40d23d0 0x7f38c41f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.310+0000 7f38ca022640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38c40d27b0 0x7f38c4148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.310+0000 7f38ca022640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f38c40cfd80 con 0x7f38c40d27b0 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.310+0000 7f38ca022640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f38c40cf860 con 0x7f38c40b9a90 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.310+0000 7f38ca022640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f38c40d0ee0 con 0x7f38c40d23d0 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.310+0000 7f38c3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38c40d27b0 0x7f38c4148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.310+0000 7f38c3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38c40d27b0 0x7f38c4148930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51478/0 (socket says 172.21.15.77:51478) 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.310+0000 7f38c3fff640 1 -- 172.21.15.77:0/3689833280 learned_addr learned my addr 172.21.15.77:0/3689833280 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.310+0000 7f38c37fe640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38c40b9a90 0x7f38c41f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.310+0000 7f38c2ffd640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38c40d23d0 0x7f38c41f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.311+0000 7f38c3fff640 1 -- 172.21.15.77:0/3689833280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38c40d23d0 msgr2=0x7f38c41f6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.311+0000 7f38c3fff640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38c40d23d0 0x7f38c41f6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.311+0000 7f38c3fff640 1 -- 172.21.15.77:0/3689833280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38c40b9a90 msgr2=0x7f38c41f69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.311+0000 7f38c3fff640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38c40b9a90 0x7f38c41f69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.311+0000 7f38c3fff640 1 -- 172.21.15.77:0/3689833280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c41f4290 con 0x7f38c40d27b0 2024-04-25T01:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.311+0000 7f38c3fff640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38c40d27b0 0x7f38c4148930 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00caf0 tx=0x7f38bc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.311+0000 7f38abfff640 1 -- 172.21.15.77:0/3689833280 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc00aca0 con 0x7f38c40d27b0 2024-04-25T01:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.311+0000 7f38abfff640 1 -- 172.21.15.77:0/3689833280 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38bc00ae40 con 0x7f38c40d27b0 2024-04-25T01:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.311+0000 7f38ca022640 1 -- 172.21.15.77:0/3689833280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c41f4530 con 0x7f38c40d27b0 2024-04-25T01:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.311+0000 7f38abfff640 1 -- 172.21.15.77:0/3689833280 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc012620 con 0x7f38c40d27b0 2024-04-25T01:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.311+0000 7f38ca022640 1 -- 172.21.15.77:0/3689833280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c41f4af0 con 0x7f38c40d27b0 2024-04-25T01:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.312+0000 7f38ca022640 1 -- 172.21.15.77:0/3689833280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c4068fb0 con 0x7f38c40d27b0 2024-04-25T01:23:15.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.312+0000 7f38abfff640 1 -- 172.21.15.77:0/3689833280 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f38bc012880 con 0x7f38c40d27b0 2024-04-25T01:23:15.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.313+0000 7f38abfff640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f38b006be30 0x7f38b006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.313+0000 7f38abfff640 1 -- 172.21.15.77:0/3689833280 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f38bc08b5e0 con 0x7f38c40d27b0 2024-04-25T01:23:15.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.313+0000 7f38c37fe640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f38b006be30 0x7f38b006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.313+0000 7f38c37fe640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f38b006be30 0x7f38b006e2f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f38b4034000 tx=0x7f38b402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:15.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.317+0000 7f38abfff640 1 -- 172.21.15.77:0/3689833280 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f38bc059760 con 0x7f38c40d27b0 2024-04-25T01:23:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.489+0000 7f38ca022640 1 -- 172.21.15.77:0/3689833280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f38c408df50 con 0x7f38c40d27b0 2024-04-25T01:23:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.490+0000 7f38abfff640 1 -- 172.21.15.77:0/3689833280 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f38bc05da60 con 0x7f38c40d27b0 2024-04-25T01:23:15.470 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:23:15.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 -- 172.21.15.77:0/3689833280 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f38b006be30 msgr2=0x7f38b006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f38b006be30 0x7f38b006e2f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f38b4034000 tx=0x7f38b402e000 comp rx=0 tx=0).stop 2024-04-25T01:23:15.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 -- 172.21.15.77:0/3689833280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38c40d27b0 msgr2=0x7f38c4148930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38c40d27b0 0x7f38c4148930 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00caf0 tx=0x7f38bc00a060 comp rx=0 tx=0).stop 2024-04-25T01:23:15.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 -- 172.21.15.77:0/3689833280 shutdown_connections 2024-04-25T01:23:15.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f38b006be30 0x7f38b006e2f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38c40d27b0 0x7f38c4148930 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38c40d23d0 0x7f38c41f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 --2- 172.21.15.77:0/3689833280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38c40b9a90 0x7f38c41f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 -- 172.21.15.77:0/3689833280 >> 172.21.15.77:0/3689833280 conn(0x7f38c40d3e30 msgr2=0x7f38c40747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:15.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 -- 172.21.15.77:0/3689833280 shutdown_connections 2024-04-25T01:23:15.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.492+0000 7f38ca022640 1 -- 172.21.15.77:0/3689833280 wait complete. 2024-04-25T01:23:15.483 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:23:15.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.602+0000 7f16b759e640 1 Processor -- start 2024-04-25T01:23:15.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.602+0000 7f16b759e640 1 -- start start 2024-04-25T01:23:15.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.603+0000 7f16b759e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b006b3c0 0x7f16b01620e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.603+0000 7f16b759e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f16b0162620 0x7f16b00b1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.603+0000 7f16b759e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16b006cb00 con 0x7f16b0161d00 2024-04-25T01:23:15.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.603+0000 7f16b759e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f16b00cff70 con 0x7f16b0162620 2024-04-25T01:23:15.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.603+0000 7f16b759e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f16b006b820 con 0x7f16b006b3c0 2024-04-25T01:23:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.603+0000 7f16b6d9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f16b0162620 0x7f16b00b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.603+0000 7f16b5d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b006b3c0 0x7f16b01620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.603+0000 7f16b6d9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f16b0162620 0x7f16b00b1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:47810/0 (socket says 172.21.15.77:47810) 2024-04-25T01:23:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.603+0000 7f16b5d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b006b3c0 0x7f16b01620e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:55180/0 (socket says 172.21.15.77:55180) 2024-04-25T01:23:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.603+0000 7f16b659c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f16b0161d00 0x7f16b00d0a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:54564/0 (socket says 172.21.15.77:54564) 2024-04-25T01:23:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.603+0000 7f16b6d9d640 1 -- 172.21.15.77:0/1092843601 learned_addr learned my addr 172.21.15.77:0/1092843601 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.604+0000 7f16b559a640 1 -- 172.21.15.77:0/1092843601 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1850783032 0 0) 0x7f16b006cb00 con 0x7f16b0161d00 2024-04-25T01:23:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.604+0000 7f16b559a640 1 -- 172.21.15.77:0/1092843601 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f169c003520 con 0x7f16b0161d00 2024-04-25T01:23:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.604+0000 7f16b6d9d640 1 -- 172.21.15.77:0/1092843601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b006b3c0 msgr2=0x7f16b01620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.604+0000 7f16b6d9d640 1 --2- 172.21.15.77:0/1092843601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b006b3c0 0x7f16b01620e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.604+0000 7f16b6d9d640 1 -- 172.21.15.77:0/1092843601 >> v1:172.21.15.121:6789/0 conn(0x7f16b0161d00 legacy=0x7f16b00d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.604+0000 7f16b6d9d640 1 -- 172.21.15.77:0/1092843601 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b006c820 con 0x7f16b0162620 2024-04-25T01:23:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.604+0000 7f16b5d9b640 1 --2- 172.21.15.77:0/1092843601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b006b3c0 0x7f16b01620e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b6d9d640 1 --2- 172.21.15.77:0/1092843601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b0162620 0x7f16b00b1210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009e50 tx=0x7f16ac02a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4ec2c1bc6da2cee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b559a640 1 -- 172.21.15.77:0/1092843601 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac036070 con 0x7f16b0162620 2024-04-25T01:23:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b559a640 1 -- 172.21.15.77:0/1092843601 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ac031600 con 0x7f16b0162620 2024-04-25T01:23:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b559a640 1 -- 172.21.15.77:0/1092843601 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac0319b0 con 0x7f16b0162620 2024-04-25T01:23:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b759e640 1 -- 172.21.15.77:0/1092843601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b0162620 msgr2=0x7f16b00b1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b759e640 1 --2- 172.21.15.77:0/1092843601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b0162620 0x7f16b00b1210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009e50 tx=0x7f16ac02a680 comp rx=0 tx=0).stop 2024-04-25T01:23:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b759e640 1 -- 172.21.15.77:0/1092843601 shutdown_connections 2024-04-25T01:23:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b759e640 1 --2- 172.21.15.77:0/1092843601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b0162620 0x7f16b00b1210 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b759e640 1 --2- 172.21.15.77:0/1092843601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b006b3c0 0x7f16b01620e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b759e640 1 -- 172.21.15.77:0/1092843601 >> 172.21.15.77:0/1092843601 conn(0x7f16b00b4f50 msgr2=0x7f16b00b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b759e640 1 -- 172.21.15.77:0/1092843601 shutdown_connections 2024-04-25T01:23:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.605+0000 7f16b759e640 1 -- 172.21.15.77:0/1092843601 wait complete. 2024-04-25T01:23:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.607+0000 7f16b759e640 1 Processor -- start 2024-04-25T01:23:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.607+0000 7f16b759e640 1 -- start start 2024-04-25T01:23:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.607+0000 7f16b759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b006b3c0 0x7f16b015d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.607+0000 7f16b759e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b0161d00 0x7f16b006ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.607+0000 7f16b759e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b0162620 0x7f16b01525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.607+0000 7f16b759e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f16b015ee20 con 0x7f16b006b3c0 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.607+0000 7f16b759e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f16b00d04f0 con 0x7f16b0161d00 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.607+0000 7f16b759e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f16b006c4b0 con 0x7f16b0162620 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b006b3c0 0x7f16b015d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b006b3c0 0x7f16b015d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51482/0 (socket says 172.21.15.77:51482) 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b659c640 1 -- 172.21.15.77:0/1425737651 learned_addr learned my addr 172.21.15.77:0/1425737651 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b6d9d640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b0162620 0x7f16b01525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b5d9b640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b0161d00 0x7f16b006ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b659c640 1 -- 172.21.15.77:0/1425737651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b0162620 msgr2=0x7f16b01525e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b659c640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b0162620 0x7f16b01525e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b659c640 1 -- 172.21.15.77:0/1425737651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b0161d00 msgr2=0x7f16b006ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b659c640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b0161d00 0x7f16b006ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b659c640 1 -- 172.21.15.77:0/1425737651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b00c95a0 con 0x7f16b006b3c0 2024-04-25T01:23:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b659c640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b006b3c0 0x7f16b015d9d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f16a400c390 tx=0x7f16a400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:15.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16a2ffd640 1 -- 172.21.15.77:0/1425737651 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a40074e0 con 0x7f16b006b3c0 2024-04-25T01:23:15.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16a2ffd640 1 -- 172.21.15.77:0/1425737651 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a40047e0 con 0x7f16b006b3c0 2024-04-25T01:23:15.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b759e640 1 -- 172.21.15.77:0/1425737651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b00c9880 con 0x7f16b006b3c0 2024-04-25T01:23:15.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16a2ffd640 1 -- 172.21.15.77:0/1425737651 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a4004bd0 con 0x7f16b006b3c0 2024-04-25T01:23:15.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.608+0000 7f16b759e640 1 -- 172.21.15.77:0/1425737651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b01f4b10 con 0x7f16b006b3c0 2024-04-25T01:23:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.609+0000 7f16b759e640 1 -- 172.21.15.77:0/1425737651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b0075350 con 0x7f16b006b3c0 2024-04-25T01:23:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.610+0000 7f16a2ffd640 1 -- 172.21.15.77:0/1425737651 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f16a401a020 con 0x7f16b006b3c0 2024-04-25T01:23:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.610+0000 7f16a2ffd640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f169406be30 0x7f169406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.610+0000 7f16a2ffd640 1 -- 172.21.15.77:0/1425737651 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f16a408a700 con 0x7f16b006b3c0 2024-04-25T01:23:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.610+0000 7f16b5d9b640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f169406be30 0x7f169406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.611+0000 7f16b5d9b640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f169406be30 0x7f169406e2f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009560 tx=0x7f16a8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:15.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.614+0000 7f16a2ffd640 1 -- 172.21.15.77:0/1425737651 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f16a4058950 con 0x7f16b006b3c0 2024-04-25T01:23:15.766 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:23:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.786+0000 7f16b759e640 1 -- 172.21.15.77:0/1425737651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f16b00b1e10 con 0x7f16b006b3c0 2024-04-25T01:23:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.786+0000 7f16a2ffd640 1 -- 172.21.15.77:0/1425737651 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7f16a405cc50 con 0x7f16b006b3c0 2024-04-25T01:23:15.768 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:23:15.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:23:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.788+0000 7f16b759e640 1 -- 172.21.15.77:0/1425737651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f169406be30 msgr2=0x7f169406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.789+0000 7f16b759e640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f169406be30 0x7f169406e2f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009560 tx=0x7f16a8009190 comp rx=0 tx=0).stop 2024-04-25T01:23:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.789+0000 7f16b759e640 1 -- 172.21.15.77:0/1425737651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b006b3c0 msgr2=0x7f16b015d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.789+0000 7f16b759e640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b006b3c0 0x7f16b015d9d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f16a400c390 tx=0x7f16a400c850 comp rx=0 tx=0).stop 2024-04-25T01:23:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.789+0000 7f16b759e640 1 -- 172.21.15.77:0/1425737651 shutdown_connections 2024-04-25T01:23:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.789+0000 7f16b759e640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f169406be30 0x7f169406e2f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.789+0000 7f16b759e640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16b0162620 0x7f16b01525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.789+0000 7f16b759e640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b0161d00 0x7f16b006ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.789+0000 7f16b759e640 1 --2- 172.21.15.77:0/1425737651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b006b3c0 0x7f16b015d9d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.789+0000 7f16b759e640 1 -- 172.21.15.77:0/1425737651 >> 172.21.15.77:0/1425737651 conn(0x7f16b00b4f50 msgr2=0x7f16b00afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.789+0000 7f16b759e640 1 -- 172.21.15.77:0/1425737651 shutdown_connections 2024-04-25T01:23:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.789+0000 7f16b759e640 1 -- 172.21.15.77:0/1425737651 wait complete. 2024-04-25T01:23:15.780 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:15.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.968+0000 7fceca5f4640 1 Processor -- start 2024-04-25T01:23:15.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.968+0000 7fceca5f4640 1 -- start start 2024-04-25T01:23:15.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.969+0000 7fceca5f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcec415e4c0 0x7fcec40d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.969+0000 7fceca5f4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40d4a10 0x7fcec40b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.969+0000 7fceca5f4640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcec40d2d40 0x7fcec40b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.969+0000 7fceca5f4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fcec40d1e90 con 0x7fcec415e4c0 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.969+0000 7fceca5f4640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcec40b54a0 con 0x7fcec40d2d40 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.969+0000 7fceca5f4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcec40d2130 con 0x7fcec40d4a10 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.969+0000 7fcec3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcec415e4c0 0x7fcec40d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.969+0000 7fcec3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcec415e4c0 0x7fcec40d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51498/0 (socket says 172.21.15.77:51498) 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.969+0000 7fcec3fff640 1 -- 172.21.15.77:0/2386150592 learned_addr learned my addr 172.21.15.77:0/2386150592 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.970+0000 7fcec37fe640 1 --2- 172.21.15.77:0/2386150592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40d4a10 0x7fcec40b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.970+0000 7fcec8b91640 1 --2- 172.21.15.77:0/2386150592 >> v2:172.21.15.121:3300/0 conn(0x7fcec40d2d40 0x7fcec40b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.970+0000 7fcec3fff640 1 -- 172.21.15.77:0/2386150592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40d4a10 msgr2=0x7fcec40b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.970+0000 7fcec3fff640 1 --2- 172.21.15.77:0/2386150592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40d4a10 0x7fcec40b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.970+0000 7fcec3fff640 1 -- 172.21.15.77:0/2386150592 >> v2:172.21.15.121:3300/0 conn(0x7fcec40d2d40 msgr2=0x7fcec40b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.970+0000 7fcec3fff640 1 --2- 172.21.15.77:0/2386150592 >> v2:172.21.15.121:3300/0 conn(0x7fcec40d2d40 0x7fcec40b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.970+0000 7fcec3fff640 1 -- 172.21.15.77:0/2386150592 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec40b5780 con 0x7fcec415e4c0 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.970+0000 7fcec3fff640 1 --2- 172.21.15.77:0/2386150592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcec415e4c0 0x7fcec40d2800 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcec408e630 tx=0x7fceb402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e6aa8c67f5faea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.970+0000 7fcec2ffd640 1 -- 172.21.15.77:0/2386150592 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4037070 con 0x7fcec415e4c0 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.971+0000 7fcec2ffd640 1 -- 172.21.15.77:0/2386150592 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb4032440 con 0x7fcec415e4c0 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.971+0000 7fcec2ffd640 1 -- 172.21.15.77:0/2386150592 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4032810 con 0x7fcec415e4c0 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.971+0000 7fceca5f4640 1 -- 172.21.15.77:0/2386150592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcec415e4c0 msgr2=0x7fcec40d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.971+0000 7fceca5f4640 1 --2- 172.21.15.77:0/2386150592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcec415e4c0 0x7fcec40d2800 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcec408e630 tx=0x7fceb402a140 comp rx=0 tx=0).stop 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.971+0000 7fceca5f4640 1 -- 172.21.15.77:0/2386150592 shutdown_connections 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.971+0000 7fceca5f4640 1 --2- 172.21.15.77:0/2386150592 >> v2:172.21.15.121:3300/0 conn(0x7fcec40d2d40 0x7fcec40b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.971+0000 7fceca5f4640 1 --2- 172.21.15.77:0/2386150592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40d4a10 0x7fcec40b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.971+0000 7fceca5f4640 1 --2- 172.21.15.77:0/2386150592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcec415e4c0 0x7fcec40d2800 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.971+0000 7fceca5f4640 1 -- 172.21.15.77:0/2386150592 >> 172.21.15.77:0/2386150592 conn(0x7fcec40d3ef0 msgr2=0x7fcec40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:15.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.971+0000 7fceca5f4640 1 -- 172.21.15.77:0/2386150592 shutdown_connections 2024-04-25T01:23:15.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.971+0000 7fceca5f4640 1 -- 172.21.15.77:0/2386150592 wait complete. 2024-04-25T01:23:15.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.972+0000 7fceca5f4640 1 Processor -- start 2024-04-25T01:23:15.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.972+0000 7fceca5f4640 1 -- start start 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fceca5f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcec40d2d40 0x7fcec415a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fceca5f4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40d4a10 0x7fcec414aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fceca5f4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcec415e4c0 0x7fcec414d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fceca5f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcec415ea20 con 0x7fcec40d2d40 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fceca5f4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcec415e8a0 con 0x7fcec415e4c0 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fceca5f4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcec40d0900 con 0x7fcec40d4a10 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fcec3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcec40d2d40 0x7fcec415a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fcec3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcec40d2d40 0x7fcec415a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51504/0 (socket says 172.21.15.77:51504) 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fcec3fff640 1 -- 172.21.15.77:0/190723440 learned_addr learned my addr 172.21.15.77:0/190723440 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fcec37fe640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40d4a10 0x7fcec414aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fcec8b91640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcec415e4c0 0x7fcec414d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fcec3fff640 1 -- 172.21.15.77:0/190723440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40d4a10 msgr2=0x7fcec414aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fcec3fff640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40d4a10 0x7fcec414aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fcec3fff640 1 -- 172.21.15.77:0/190723440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcec415e4c0 msgr2=0x7fcec414d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.973+0000 7fcec3fff640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcec415e4c0 0x7fcec414d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.974+0000 7fcec3fff640 1 -- 172.21.15.77:0/190723440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec4159820 con 0x7fcec40d2d40 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.974+0000 7fcec3fff640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcec40d2d40 0x7fcec415a7c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fceb4000c00 tx=0x7fceb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.974+0000 7fcec0ff9640 1 -- 172.21.15.77:0/190723440 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4041070 con 0x7fcec40d2d40 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.974+0000 7fcec0ff9640 1 -- 172.21.15.77:0/190723440 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb403c870 con 0x7fcec40d2d40 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.974+0000 7fcec0ff9640 1 -- 172.21.15.77:0/190723440 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4037040 con 0x7fcec40d2d40 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.974+0000 7fceca5f4640 1 -- 172.21.15.77:0/190723440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec415aec0 con 0x7fcec40d2d40 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.974+0000 7fceca5f4640 1 -- 172.21.15.77:0/190723440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec415b480 con 0x7fcec40d2d40 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.975+0000 7fceca5f4640 1 -- 172.21.15.77:0/190723440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec406ce50 con 0x7fcec40d2d40 2024-04-25T01:23:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.975+0000 7fcec0ff9640 1 -- 172.21.15.77:0/190723440 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fceb4044070 con 0x7fcec40d2d40 2024-04-25T01:23:15.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.975+0000 7fcec0ff9640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fceb006be30 0x7fceb006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.976+0000 7fcec0ff9640 1 -- 172.21.15.77:0/190723440 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7fceb40abb90 con 0x7fcec40d2d40 2024-04-25T01:23:15.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.976+0000 7fcec37fe640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fceb006be30 0x7fceb006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.976+0000 7fcec37fe640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fceb006be30 0x7fceb006e2f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcec40d4470 tx=0x7fceb8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:15.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:15.980+0000 7fcec0ff9640 1 -- 172.21.15.77:0/190723440 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcec406ce50 con 0x7fcec40d2d40 2024-04-25T01:23:16.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.190+0000 7fceca5f4640 1 -- 172.21.15.77:0/190723440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcec4068fb0 con 0x7fcec40d2d40 2024-04-25T01:23:16.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.191+0000 7fcec0ff9640 1 -- 172.21.15.77:0/190723440 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5188 (secure 0 0 0) 0x7fceb407a4f0 con 0x7fcec40d2d40 2024-04-25T01:23:16.171 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:16.171 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2823778882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2823778882},{"type":"v1","addr":"172.21.15.77:6837","nonce":2823778882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.066906+0000","modified":"2024-04-25T01:23:09.251757+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5355":{"gid":5355,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6835/3139470828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3139470828},{"type":"v1","addr":"172.21.15.77:6835","nonce":3139470828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5355,"qdb_cluster":[5355]},"id":2}]} 2024-04-25T01:23:16.172 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-04-25T01:23:16.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 -- 172.21.15.77:0/190723440 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fceb006be30 msgr2=0x7fceb006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fceb006be30 0x7fceb006e2f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcec40d4470 tx=0x7fceb8008040 comp rx=0 tx=0).stop 2024-04-25T01:23:16.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 -- 172.21.15.77:0/190723440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcec40d2d40 msgr2=0x7fcec415a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcec40d2d40 0x7fcec415a7c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fceb4000c00 tx=0x7fceb40046c0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 -- 172.21.15.77:0/190723440 shutdown_connections 2024-04-25T01:23:16.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fceb006be30 0x7fceb006e2f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcec415e4c0 0x7fcec414d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40d4a10 0x7fcec414aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 --2- 172.21.15.77:0/190723440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcec40d2d40 0x7fcec415a7c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 -- 172.21.15.77:0/190723440 >> 172.21.15.77:0/190723440 conn(0x7fcec40d3ef0 msgr2=0x7fcec40747f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:16.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 -- 172.21.15.77:0/190723440 shutdown_connections 2024-04-25T01:23:16.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.193+0000 7fceca5f4640 1 -- 172.21.15.77:0/190723440 wait complete. 2024-04-25T01:23:16.184 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:16.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.301+0000 7fc100e88640 1 Processor -- start 2024-04-25T01:23:16.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.301+0000 7fc100e88640 1 -- start start 2024-04-25T01:23:16.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.302+0000 7fc100e88640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc0fc15fe10 0x7fc0fc06d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.302+0000 7fc100e88640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0fc0b4240 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.302+0000 7fc100e88640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0fc0b39a0 con 0x7fc0fc145590 2024-04-25T01:23:16.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.302+0000 7fc100e88640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fc0fc160830 con 0x7fc0fc15fe10 2024-04-25T01:23:16.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.303+0000 7fc0fad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc0fc15fe10 0x7fc0fc06d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.303+0000 7fc0f9d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc0fc15f9b0 0x7fc0fc06d270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44938/0 (socket says 172.21.15.77:44938) 2024-04-25T01:23:16.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.303+0000 7fc0f9d9b640 1 -- 172.21.15.77:0/2103059177 learned_addr learned my addr 172.21.15.77:0/2103059177 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.303+0000 7fc0fad9d640 1 -- 172.21.15.77:0/2103059177 >> v1:172.21.15.121:6789/0 conn(0x7fc0fc145590 legacy=0x7fc0fc0b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.303+0000 7fc0f959a640 1 -- 172.21.15.77:0/2103059177 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3240924195 0 0) 0x7fc0fc0b4240 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.303+0000 7fc0fad9d640 1 -- 172.21.15.77:0/2103059177 >> v1:172.21.15.77:6789/0 conn(0x7fc0fc15f9b0 legacy=0x7fc0fc06d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.303+0000 7fc0fad9d640 1 -- 172.21.15.77:0/2103059177 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0fc0b3c80 con 0x7fc0fc15fe10 2024-04-25T01:23:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.304+0000 7fc0fad9d640 1 --2- 172.21.15.77:0/2103059177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0fc15fe10 0x7fc0fc06d980 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4009d90 tx=0x7fc0f402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6e140e4c18062a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.304+0000 7fc0f959a640 1 -- 172.21.15.77:0/2103059177 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f4007740 con 0x7fc0fc15fe10 2024-04-25T01:23:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.304+0000 7fc0f959a640 1 -- 172.21.15.77:0/2103059177 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0f40078e0 con 0x7fc0fc15fe10 2024-04-25T01:23:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.304+0000 7fc0f959a640 1 -- 172.21.15.77:0/2103059177 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f4007cb0 con 0x7fc0fc15fe10 2024-04-25T01:23:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.304+0000 7fc100e88640 1 -- 172.21.15.77:0/2103059177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0fc15fe10 msgr2=0x7fc0fc06d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.304+0000 7fc100e88640 1 --2- 172.21.15.77:0/2103059177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0fc15fe10 0x7fc0fc06d980 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4009d90 tx=0x7fc0f402a6f0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.304+0000 7fc100e88640 1 -- 172.21.15.77:0/2103059177 shutdown_connections 2024-04-25T01:23:16.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.304+0000 7fc100e88640 1 --2- 172.21.15.77:0/2103059177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0fc15fe10 0x7fc0fc06d980 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.304+0000 7fc100e88640 1 -- 172.21.15.77:0/2103059177 >> 172.21.15.77:0/2103059177 conn(0x7fc0fc0b1820 msgr2=0x7fc0fc0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:16.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.304+0000 7fc100e88640 1 -- 172.21.15.77:0/2103059177 shutdown_connections 2024-04-25T01:23:16.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.304+0000 7fc100e88640 1 -- 172.21.15.77:0/2103059177 wait complete. 2024-04-25T01:23:16.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.306+0000 7fc100e88640 1 Processor -- start 2024-04-25T01:23:16.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.306+0000 7fc100e88640 1 -- start start 2024-04-25T01:23:16.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.306+0000 7fc100e88640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0fc145590 0x7fc0fc1f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.306+0000 7fc100e88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0fc15f9b0 0x7fc0fc143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.306+0000 7fc100e88640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0fc15fe10 0x7fc0fc143870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.306+0000 7fc100e88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc160280 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.306+0000 7fc100e88640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc06cde0 con 0x7fc0fc15fe10 2024-04-25T01:23:16.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.306+0000 7fc100e88640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc0fc145970 con 0x7fc0fc145590 2024-04-25T01:23:16.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.306+0000 7fc0f9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0fc15f9b0 0x7fc0fc143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0f9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0fc15f9b0 0x7fc0fc143330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51520/0 (socket says 172.21.15.77:51520) 2024-04-25T01:23:16.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0f9d9b640 1 -- 172.21.15.77:0/3218378947 learned_addr learned my addr 172.21.15.77:0/3218378947 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:16.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0fad9d640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0fc15fe10 0x7fc0fc143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0fa59c640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0fc145590 0x7fc0fc1f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0f9d9b640 1 -- 172.21.15.77:0/3218378947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0fc145590 msgr2=0x7fc0fc1f16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0f9d9b640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0fc145590 0x7fc0fc1f16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0f9d9b640 1 -- 172.21.15.77:0/3218378947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0fc15fe10 msgr2=0x7fc0fc143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0f9d9b640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0fc15fe10 0x7fc0fc143870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0f9d9b640 1 -- 172.21.15.77:0/3218378947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0fc15d120 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0f9d9b640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0fc15f9b0 0x7fc0fc143330 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00c4a0 tx=0x7fc0ec00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:16.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0eaffd640 1 -- 172.21.15.77:0/3218378947 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec018070 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0eaffd640 1 -- 172.21.15.77:0/3218378947 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0ec0074e0 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc100e88640 1 -- 172.21.15.77:0/3218378947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0fc1f55b0 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc0eaffd640 1 -- 172.21.15.77:0/3218378947 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec004a20 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.307+0000 7fc100e88640 1 -- 172.21.15.77:0/3218378947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0fc1f5b70 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.308+0000 7fc100e88640 1 -- 172.21.15.77:0/3218378947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0fc1f53a0 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.308+0000 7fc0eaffd640 1 -- 172.21.15.77:0/3218378947 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc0ec01b070 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.309+0000 7fc0eaffd640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0dc06be30 0x7fc0dc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.309+0000 7fc0eaffd640 1 -- 172.21.15.77:0/3218378947 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7fc0ec08a330 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.309+0000 7fc0fa59c640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0dc06be30 0x7fc0dc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.309+0000 7fc0fa59c640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0dc06be30 0x7fc0dc06e2f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0000f60 tx=0x7fc0f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:16.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.313+0000 7fc0eaffd640 1 -- 172.21.15.77:0/3218378947 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc0ec058b90 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.488+0000 7fc100e88640 1 -- 172.21.15.77:0/3218378947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0fc068fb0 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.489+0000 7fc0eaffd640 1 -- 172.21.15.77:0/3218378947 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+17071 (secure 0 0 0) 0x7fc0ec058490 con 0x7fc0fc15f9b0 2024-04-25T01:23:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:16.470 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":39,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:23:06.238842+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.457887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.029883+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6837/2944320661":"2024-04-26T01:23:06.238820+0000","172.21.15.77:6836/2944320661":"2024-04-26T01:23:06.238820+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 -- 172.21.15.77:0/3218378947 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0dc06be30 msgr2=0x7fc0dc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0dc06be30 0x7fc0dc06e2f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0000f60 tx=0x7fc0f0009040 comp rx=0 tx=0).stop 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 -- 172.21.15.77:0/3218378947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0fc15f9b0 msgr2=0x7fc0fc143330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0fc15f9b0 0x7fc0fc143330 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00c4a0 tx=0x7fc0ec00c960 comp rx=0 tx=0).stop 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 -- 172.21.15.77:0/3218378947 shutdown_connections 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0dc06be30 0x7fc0dc06e2f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0fc15f9b0 0x7fc0fc143330 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0fc15fe10 0x7fc0fc143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 --2- 172.21.15.77:0/3218378947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0fc145590 0x7fc0fc1f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 -- 172.21.15.77:0/3218378947 >> 172.21.15.77:0/3218378947 conn(0x7fc0fc0b1820 msgr2=0x7fc0fc0af870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 -- 172.21.15.77:0/3218378947 shutdown_connections 2024-04-25T01:23:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.491+0000 7fc100e88640 1 -- 172.21.15.77:0/3218378947 wait complete. 2024-04-25T01:23:16.482 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T01:23:16.482 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:16.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.598+0000 7f994b44d640 1 Processor -- start 2024-04-25T01:23:16.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.598+0000 7f994b44d640 1 -- start start 2024-04-25T01:23:16.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.599+0000 7f994b44d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99440d4a10 0x7f99440cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.599+0000 7f994b44d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99440b3ee0 0x7f99440b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.599+0000 7f994b44d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99440d1e90 con 0x7f994415e4c0 2024-04-25T01:23:16.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.599+0000 7f994b44d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f99440b54a0 con 0x7f99440b3ee0 2024-04-25T01:23:16.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.599+0000 7f994b44d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f99440d2130 con 0x7f99440d4a10 2024-04-25T01:23:16.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.599+0000 7f99499ea640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99440b3ee0 0x7f99440b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.599+0000 7f99491e9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f994415e4c0 0x7f99440d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44228/0 (socket says 172.21.15.77:44228) 2024-04-25T01:23:16.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.599+0000 7f99491e9640 1 -- 172.21.15.77:0/2268308573 learned_addr learned my addr 172.21.15.77:0/2268308573 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:16.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.600+0000 7f99489e8640 1 --2- 172.21.15.77:0/2268308573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99440d4a10 0x7f99440cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.600+0000 7f99499ea640 1 -- 172.21.15.77:0/2268308573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99440d4a10 msgr2=0x7f99440cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:16.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.600+0000 7f99499ea640 1 --2- 172.21.15.77:0/2268308573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99440d4a10 0x7f99440cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.600+0000 7f99499ea640 1 -- 172.21.15.77:0/2268308573 >> v1:172.21.15.77:6789/0 conn(0x7f994415e4c0 legacy=0x7f99440d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.600+0000 7f9943fff640 1 -- 172.21.15.77:0/2268308573 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3650061556 0 0) 0x7f99440d1e90 con 0x7f994415e4c0 2024-04-25T01:23:16.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.600+0000 7f99499ea640 1 -- 172.21.15.77:0/2268308573 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99440b5780 con 0x7f99440b3ee0 2024-04-25T01:23:16.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.600+0000 7f99499ea640 1 --2- 172.21.15.77:0/2268308573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99440b3ee0 0x7f99440b4390 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f993c009d90 tx=0x7f993c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c1d22a7095f7d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:16.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.600+0000 7f9943fff640 1 -- 172.21.15.77:0/2268308573 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c031600 con 0x7f99440b3ee0 2024-04-25T01:23:16.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.601+0000 7f9943fff640 1 -- 172.21.15.77:0/2268308573 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f993c0317a0 con 0x7f99440b3ee0 2024-04-25T01:23:16.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.601+0000 7f9943fff640 1 -- 172.21.15.77:0/2268308573 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c031b70 con 0x7f99440b3ee0 2024-04-25T01:23:16.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.601+0000 7f994b44d640 1 -- 172.21.15.77:0/2268308573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99440b3ee0 msgr2=0x7f99440b4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.601+0000 7f994b44d640 1 --2- 172.21.15.77:0/2268308573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99440b3ee0 0x7f99440b4390 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f993c009d90 tx=0x7f993c02a5d0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.601+0000 7f994b44d640 1 -- 172.21.15.77:0/2268308573 shutdown_connections 2024-04-25T01:23:16.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.601+0000 7f994b44d640 1 --2- 172.21.15.77:0/2268308573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99440b3ee0 0x7f99440b4390 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.601+0000 7f994b44d640 1 --2- 172.21.15.77:0/2268308573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99440d4a10 0x7f99440cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.601+0000 7f994b44d640 1 -- 172.21.15.77:0/2268308573 >> 172.21.15.77:0/2268308573 conn(0x7f99440d3ef0 msgr2=0x7f99440d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:16.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.601+0000 7f994b44d640 1 -- 172.21.15.77:0/2268308573 shutdown_connections 2024-04-25T01:23:16.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.601+0000 7f994b44d640 1 -- 172.21.15.77:0/2268308573 wait complete. 2024-04-25T01:23:16.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.602+0000 7f994b44d640 1 Processor -- start 2024-04-25T01:23:16.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.602+0000 7f994b44d640 1 -- start start 2024-04-25T01:23:16.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f994b44d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99440b3ee0 0x7f994415aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f994b44d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99440d4a10 0x7f99441f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f994b44d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994415e4c0 0x7f9944159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f994b44d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f99440b30a0 con 0x7f994415e4c0 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f994b44d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f99440b2f20 con 0x7f99440d4a10 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f994b44d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f994415e8a0 con 0x7f99440b3ee0 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f99499ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994415e4c0 0x7f9944159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f99499ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994415e4c0 0x7f9944159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54420/0 (socket says 172.21.15.77:54420) 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f99499ea640 1 -- 172.21.15.77:0/1801623584 learned_addr learned my addr 172.21.15.77:0/1801623584 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f99489e8640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99440d4a10 0x7f99441f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f99491e9640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99440b3ee0 0x7f994415aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f99499ea640 1 -- 172.21.15.77:0/1801623584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99440b3ee0 msgr2=0x7f994415aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f99499ea640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99440b3ee0 0x7f994415aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f99499ea640 1 -- 172.21.15.77:0/1801623584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99440d4a10 msgr2=0x7f99441f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.603+0000 7f99499ea640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99440d4a10 0x7f99441f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.604+0000 7f99499ea640 1 -- 172.21.15.77:0/1801623584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99440c7ac0 con 0x7f994415e4c0 2024-04-25T01:23:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.604+0000 7f99491e9640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99440b3ee0 0x7f994415aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:16.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.604+0000 7f99499ea640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994415e4c0 0x7f9944159600 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f993c02abc0 tx=0x7f993c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:16.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.604+0000 7f9941ffb640 1 -- 172.21.15.77:0/1801623584 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c040070 con 0x7f994415e4c0 2024-04-25T01:23:16.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.604+0000 7f9941ffb640 1 -- 172.21.15.77:0/1801623584 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f993c030b60 con 0x7f994415e4c0 2024-04-25T01:23:16.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.604+0000 7f994b44d640 1 -- 172.21.15.77:0/1801623584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9944159e50 con 0x7f994415e4c0 2024-04-25T01:23:16.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.604+0000 7f9941ffb640 1 -- 172.21.15.77:0/1801623584 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c045460 con 0x7f994415e4c0 2024-04-25T01:23:16.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.604+0000 7f994b44d640 1 -- 172.21.15.77:0/1801623584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f994415a410 con 0x7f994415e4c0 2024-04-25T01:23:16.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.605+0000 7f994b44d640 1 -- 172.21.15.77:0/1801623584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9944068fb0 con 0x7f994415e4c0 2024-04-25T01:23:16.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.605+0000 7f9941ffb640 1 -- 172.21.15.77:0/1801623584 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f993c039040 con 0x7f994415e4c0 2024-04-25T01:23:16.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.606+0000 7f9941ffb640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f993006be30 0x7f993006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.606+0000 7f9941ffb640 1 -- 172.21.15.77:0/1801623584 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f993c0ac7b0 con 0x7f994415e4c0 2024-04-25T01:23:16.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.606+0000 7f99491e9640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f993006be30 0x7f993006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.606+0000 7f99491e9640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f993006be30 0x7f993006e2f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9934000f60 tx=0x7f9934007ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:16.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.610+0000 7f9941ffb640 1 -- 172.21.15.77:0/1801623584 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f993c07b040 con 0x7f994415e4c0 2024-04-25T01:23:16.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.818+0000 7f994b44d640 1 -- 172.21.15.77:0/1801623584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99440d37d0 con 0x7f994415e4c0 2024-04-25T01:23:16.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.818+0000 7f9941ffb640 1 -- 172.21.15.77:0/1801623584 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5188 (secure 0 0 0) 0x7f993c07a940 con 0x7f994415e4c0 2024-04-25T01:23:16.799 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:16.799 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2823778882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2823778882},{"type":"v1","addr":"172.21.15.77:6837","nonce":2823778882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.066906+0000","modified":"2024-04-25T01:23:09.251757+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5355":{"gid":5355,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6835/3139470828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3139470828},{"type":"v1","addr":"172.21.15.77:6835","nonce":3139470828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5355,"qdb_cluster":[5355]},"id":2}]} 2024-04-25T01:23:16.799 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-04-25T01:23:16.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 -- 172.21.15.77:0/1801623584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f993006be30 msgr2=0x7f993006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f993006be30 0x7f993006e2f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9934000f60 tx=0x7f9934007ca0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 -- 172.21.15.77:0/1801623584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994415e4c0 msgr2=0x7f9944159600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994415e4c0 0x7f9944159600 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f993c02abc0 tx=0x7f993c004a40 comp rx=0 tx=0).stop 2024-04-25T01:23:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 -- 172.21.15.77:0/1801623584 shutdown_connections 2024-04-25T01:23:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f993006be30 0x7f993006e2f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994415e4c0 0x7f9944159600 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99440d4a10 0x7f99441f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 --2- 172.21.15.77:0/1801623584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99440b3ee0 0x7f994415aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 -- 172.21.15.77:0/1801623584 >> 172.21.15.77:0/1801623584 conn(0x7f99440d3ef0 msgr2=0x7f99440d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 -- 172.21.15.77:0/1801623584 shutdown_connections 2024-04-25T01:23:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.820+0000 7f994b44d640 1 -- 172.21.15.77:0/1801623584 wait complete. 2024-04-25T01:23:16.811 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:16.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.927+0000 7ff31619e640 1 Processor -- start 2024-04-25T01:23:16.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.927+0000 7ff31619e640 1 -- start start 2024-04-25T01:23:16.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff31619e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31015e2f0 0x7ff3100b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff31619e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3100b2da0 con 0x7ff31006ce50 2024-04-25T01:23:16.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff31619e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3100d0520 con 0x7ff31015e6d0 2024-04-25T01:23:16.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff31619e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff3100d0800 con 0x7ff31015e2f0 2024-04-25T01:23:16.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff31599d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff31006ce50 0x7ff3100b6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44232/0 (socket says 172.21.15.77:44232) 2024-04-25T01:23:16.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff31599d640 1 -- 172.21.15.77:0/1526086084 learned_addr learned my addr 172.21.15.77:0/1526086084 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:16.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff31519c640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31015e2f0 0x7ff3100b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3331549318 0 0) 0x7ff3100b2da0 con 0x7ff31006ce50 2024-04-25T01:23:16.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2fc003520 con 0x7ff31006ce50 2024-04-25T01:23:16.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2966363755 0 0) 0x7ff3100d0520 con 0x7ff31015e6d0 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3100b2da0 con 0x7ff31015e6d0 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3722760720 0 0) 0x7ff2fc003520 con 0x7ff31006ce50 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31015e2f0 msgr2=0x7ff3100b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31015e2f0 0x7ff3100b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 >> v1:172.21.15.121:6789/0 conn(0x7ff31015e6d0 legacy=0x7ff31006c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3100b5a90 con 0x7ff31006ce50 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7ff3080019f0 con 0x7ff31006ce50 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 >> v1:172.21.15.77:6789/0 conn(0x7ff31006ce50 legacy=0x7ff3100b6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 0x7ff2fc003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2fc004800 0x7ff2fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31015e2f0 0x7ff2fc005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff3100d0800 con 0x7ff2fc003b70 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc00bb80 con 0x7ff2fc004800 2024-04-25T01:23:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.928+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff2fc00bd80 con 0x7ff31015e2f0 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.929+0000 7ff31519c640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 0x7ff2fc003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.929+0000 7ff31599d640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31015e2f0 0x7ff2fc005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.929+0000 7ff31499b640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2fc004800 0x7ff2fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.929+0000 7ff31519c640 1 -- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31015e2f0 msgr2=0x7ff2fc005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.929+0000 7ff31519c640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff31015e2f0 0x7ff2fc005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.929+0000 7ff31519c640 1 -- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2fc004800 msgr2=0x7ff2fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.929+0000 7ff31519c640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2fc004800 0x7ff2fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.929+0000 7ff31519c640 1 -- 172.21.15.77:0/1526086084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2fc003520 con 0x7ff2fc003b70 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.929+0000 7ff31499b640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2fc004800 0x7ff2fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.929+0000 7ff31519c640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 0x7ff2fc003f70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009ac0 tx=0x7ff2fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dba3e5b85c091bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.930+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c036070 con 0x7ff2fc003b70 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.930+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff30c031400 con 0x7ff2fc003b70 2024-04-25T01:23:16.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.930+0000 7ff307fff640 1 -- 172.21.15.77:0/1526086084 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c009690 con 0x7ff2fc003b70 2024-04-25T01:23:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.930+0000 7ff31619e640 1 -- 172.21.15.77:0/1526086084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 msgr2=0x7ff2fc003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.930+0000 7ff31619e640 1 --2- 172.21.15.77:0/1526086084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 0x7ff2fc003f70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009ac0 tx=0x7ff2fc005830 comp rx=0 tx=0).stop 2024-04-25T01:23:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.930+0000 7ff31599d640 1 -- 172.21.15.77:0/1526086084 reap_dead start 2024-04-25T01:23:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.930+0000 7ff31619e640 1 -- 172.21.15.77:0/1526086084 shutdown_connections 2024-04-25T01:23:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.930+0000 7ff31619e640 1 -- 172.21.15.77:0/1526086084 >> 172.21.15.77:0/1526086084 conn(0x7ff3100d1b90 msgr2=0x7ff3100d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.930+0000 7ff31619e640 1 -- 172.21.15.77:0/1526086084 shutdown_connections 2024-04-25T01:23:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.930+0000 7ff31619e640 1 -- 172.21.15.77:0/1526086084 wait complete. 2024-04-25T01:23:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.931+0000 7ff31619e640 1 Processor -- start 2024-04-25T01:23:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31619e640 1 -- start start 2024-04-25T01:23:16.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31619e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 0x7ff31015a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31619e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3100b3f60 0x7ff3100b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31619e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3100c5810 0x7ff3100c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31619e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff3100d0c60 con 0x7ff2fc003b70 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31619e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff3100d0ae0 con 0x7ff3100c5810 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31619e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff3100b3020 con 0x7ff3100b3f60 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31519c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 0x7ff31015a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31519c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 0x7ff31015a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54444/0 (socket says 172.21.15.77:54444) 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31519c640 1 -- 172.21.15.77:0/1378949340 learned_addr learned my addr 172.21.15.77:0/1378949340 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31499b640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3100b3f60 0x7ff3100b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.932+0000 7ff31599d640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3100c5810 0x7ff3100c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.933+0000 7ff31519c640 1 -- 172.21.15.77:0/1378949340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3100b3f60 msgr2=0x7ff3100b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.933+0000 7ff31519c640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3100b3f60 0x7ff3100b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.933+0000 7ff31519c640 1 -- 172.21.15.77:0/1378949340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3100c5810 msgr2=0x7ff3100c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.933+0000 7ff31519c640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3100c5810 0x7ff3100c5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.933+0000 7ff31519c640 1 -- 172.21.15.77:0/1378949340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3100c6580 con 0x7ff2fc003b70 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.933+0000 7ff31519c640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 0x7ff31015a420 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009be0 tx=0x7ff30c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.933+0000 7ff305ffb640 1 -- 172.21.15.77:0/1378949340 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c040020 con 0x7ff2fc003b70 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.933+0000 7ff305ffb640 1 -- 172.21.15.77:0/1378949340 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff30c003b40 con 0x7ff2fc003b70 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.933+0000 7ff31619e640 1 -- 172.21.15.77:0/1378949340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3100c6860 con 0x7ff2fc003b70 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.933+0000 7ff305ffb640 1 -- 172.21.15.77:0/1378949340 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c036040 con 0x7ff2fc003b70 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.933+0000 7ff31619e640 1 -- 172.21.15.77:0/1378949340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3100c6e20 con 0x7ff2fc003b70 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.934+0000 7ff31619e640 1 -- 172.21.15.77:0/1378949340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff310075350 con 0x7ff2fc003b70 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.934+0000 7ff305ffb640 1 -- 172.21.15.77:0/1378949340 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff30c043020 con 0x7ff2fc003b70 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.935+0000 7ff305ffb640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2f406be30 0x7ff2f406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:16.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.935+0000 7ff305ffb640 1 -- 172.21.15.77:0/1378949340 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7ff30c0ab2b0 con 0x7ff2fc003b70 2024-04-25T01:23:16.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.935+0000 7ff31499b640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2f406be30 0x7ff2f406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:16.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.935+0000 7ff31499b640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2f406be30 0x7ff2f406e2f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff300007fd0 tx=0x7ff300009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:16.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:16.939+0000 7ff305ffb640 1 -- 172.21.15.77:0/1378949340 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff30c04a070 con 0x7ff2fc003b70 2024-04-25T01:23:17.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.146+0000 7ff31619e640 1 -- 172.21.15.77:0/1378949340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff31006f1d0 con 0x7ff2fc003b70 2024-04-25T01:23:17.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.147+0000 7ff305ffb640 1 -- 172.21.15.77:0/1378949340 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5188 (secure 0 0 0) 0x7ff30c0044e0 con 0x7ff2fc003b70 2024-04-25T01:23:17.127 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:17.127 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2823778882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2823778882},{"type":"v1","addr":"172.21.15.77:6837","nonce":2823778882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.066906+0000","modified":"2024-04-25T01:23:09.251757+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5355":{"gid":5355,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6835/3139470828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3139470828},{"type":"v1","addr":"172.21.15.77:6835","nonce":3139470828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5355,"qdb_cluster":[5355]},"id":2}]} 2024-04-25T01:23:17.127 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-04-25T01:23:17.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 -- 172.21.15.77:0/1378949340 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2f406be30 msgr2=0x7ff2f406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2f406be30 0x7ff2f406e2f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff300007fd0 tx=0x7ff300009040 comp rx=0 tx=0).stop 2024-04-25T01:23:17.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 -- 172.21.15.77:0/1378949340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 msgr2=0x7ff31015a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 0x7ff31015a420 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009be0 tx=0x7ff30c004730 comp rx=0 tx=0).stop 2024-04-25T01:23:17.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 -- 172.21.15.77:0/1378949340 shutdown_connections 2024-04-25T01:23:17.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff2f406be30 0x7ff2f406e2f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3100c5810 0x7ff3100c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3100b3f60 0x7ff3100b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 --2- 172.21.15.77:0/1378949340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2fc003b70 0x7ff31015a420 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 -- 172.21.15.77:0/1378949340 >> 172.21.15.77:0/1378949340 conn(0x7ff3100d1b90 msgr2=0x7ff31015c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:17.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 -- 172.21.15.77:0/1378949340 shutdown_connections 2024-04-25T01:23:17.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.149+0000 7ff31619e640 1 -- 172.21.15.77:0/1378949340 wait complete. 2024-04-25T01:23:17.140 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:17.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.256+0000 7fc825790640 1 Processor -- start 2024-04-25T01:23:17.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.256+0000 7fc825790640 1 -- start start 2024-04-25T01:23:17.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.257+0000 7fc825790640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8200ce500 0x7fc8200ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.257+0000 7fc825790640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc82006feb0 0x7fc820162550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.257+0000 7fc825790640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8200cee20 con 0x7fc82006fa50 2024-04-25T01:23:17.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.257+0000 7fc825790640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc8200d0f40 con 0x7fc82006feb0 2024-04-25T01:23:17.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.257+0000 7fc825790640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc8200d0330 con 0x7fc8200ce500 2024-04-25T01:23:17.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.257+0000 7fc81f7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc82006feb0 0x7fc820162550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.257+0000 7fc81e7fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc82006fa50 0x7fc820161e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:42654/0 (socket says 172.21.15.77:42654) 2024-04-25T01:23:17.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.257+0000 7fc81e7fc640 1 -- 172.21.15.77:0/3835138351 learned_addr learned my addr 172.21.15.77:0/3835138351 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:17.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.257+0000 7fc81effd640 1 --2- 172.21.15.77:0/3835138351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8200ce500 0x7fc8200ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.258+0000 7fc81f7fe640 1 -- 172.21.15.77:0/3835138351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8200ce500 msgr2=0x7fc8200ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:17.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.258+0000 7fc81f7fe640 1 --2- 172.21.15.77:0/3835138351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8200ce500 0x7fc8200ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.258+0000 7fc81f7fe640 1 -- 172.21.15.77:0/3835138351 >> v1:172.21.15.121:6789/0 conn(0x7fc82006fa50 legacy=0x7fc820161e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.258+0000 7fc81f7fe640 1 -- 172.21.15.77:0/3835138351 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8200d0610 con 0x7fc82006feb0 2024-04-25T01:23:17.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.258+0000 7fc81dffb640 1 -- 172.21.15.77:0/3835138351 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737488609 0 0) 0x7fc8200cee20 con 0x7fc82006fa50 2024-04-25T01:23:17.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.258+0000 7fc81effd640 1 --2- 172.21.15.77:0/3835138351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8200ce500 0x7fc8200ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:17.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.258+0000 7fc81f7fe640 1 --2- 172.21.15.77:0/3835138351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc82006feb0 0x7fc820162550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc8180097d0 tx=0x7fc81802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a44f6af622d8704 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:17.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.259+0000 7fc81dffb640 1 -- 172.21.15.77:0/3835138351 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc818037070 con 0x7fc82006feb0 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.259+0000 7fc81dffb640 1 -- 172.21.15.77:0/3835138351 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc818032440 con 0x7fc82006feb0 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.259+0000 7fc81dffb640 1 -- 172.21.15.77:0/3835138351 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc818032810 con 0x7fc82006feb0 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.259+0000 7fc825790640 1 -- 172.21.15.77:0/3835138351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc82006feb0 msgr2=0x7fc820162550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.259+0000 7fc825790640 1 --2- 172.21.15.77:0/3835138351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc82006feb0 0x7fc820162550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc8180097d0 tx=0x7fc81802a1c0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.259+0000 7fc825790640 1 -- 172.21.15.77:0/3835138351 shutdown_connections 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.259+0000 7fc825790640 1 --2- 172.21.15.77:0/3835138351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc82006feb0 0x7fc820162550 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.259+0000 7fc825790640 1 --2- 172.21.15.77:0/3835138351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8200ce500 0x7fc8200ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.259+0000 7fc825790640 1 -- 172.21.15.77:0/3835138351 >> 172.21.15.77:0/3835138351 conn(0x7fc8200d1c80 msgr2=0x7fc8200d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.259+0000 7fc825790640 1 -- 172.21.15.77:0/3835138351 shutdown_connections 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.259+0000 7fc825790640 1 -- 172.21.15.77:0/3835138351 wait complete. 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.260+0000 7fc825790640 1 Processor -- start 2024-04-25T01:23:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.260+0000 7fc825790640 1 -- start start 2024-04-25T01:23:17.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.261+0000 7fc825790640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc82006fa50 0x7fc820156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.261+0000 7fc825790640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc82006feb0 0x7fc8201f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.261+0000 7fc825790640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8200ce500 0x7fc8200b2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.261+0000 7fc825790640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc8200b1390 con 0x7fc82006feb0 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.261+0000 7fc825790640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc8200d0bd0 con 0x7fc82006fa50 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.261+0000 7fc825790640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc8200b1510 con 0x7fc8200ce500 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.261+0000 7fc81e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc82006feb0 0x7fc8201f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.261+0000 7fc81e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc82006feb0 0x7fc8201f3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54446/0 (socket says 172.21.15.77:54446) 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.261+0000 7fc81e7fc640 1 -- 172.21.15.77:0/2272520651 learned_addr learned my addr 172.21.15.77:0/2272520651 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.261+0000 7fc81effd640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc82006fa50 0x7fc820156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.261+0000 7fc81f7fe640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8200ce500 0x7fc8200b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.262+0000 7fc81e7fc640 1 -- 172.21.15.77:0/2272520651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8200ce500 msgr2=0x7fc8200b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.262+0000 7fc81e7fc640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8200ce500 0x7fc8200b2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.262+0000 7fc81e7fc640 1 -- 172.21.15.77:0/2272520651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc82006fa50 msgr2=0x7fc820156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.262+0000 7fc81e7fc640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc82006fa50 0x7fc820156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.262+0000 7fc81e7fc640 1 -- 172.21.15.77:0/2272520651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8201f4ba0 con 0x7fc82006feb0 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.262+0000 7fc81e7fc640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc82006feb0 0x7fc8201f3c40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc81400c500 tx=0x7fc81400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.262+0000 7fc8077fe640 1 -- 172.21.15.77:0/2272520651 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc814018070 con 0x7fc82006feb0 2024-04-25T01:23:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.262+0000 7fc8077fe640 1 -- 172.21.15.77:0/2272520651 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc814007580 con 0x7fc82006feb0 2024-04-25T01:23:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.262+0000 7fc825790640 1 -- 172.21.15.77:0/2272520651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8201f45e0 con 0x7fc82006feb0 2024-04-25T01:23:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.262+0000 7fc8077fe640 1 -- 172.21.15.77:0/2272520651 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc814004a20 con 0x7fc82006feb0 2024-04-25T01:23:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.262+0000 7fc825790640 1 -- 172.21.15.77:0/2272520651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8200c70d0 con 0x7fc82006feb0 2024-04-25T01:23:17.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.263+0000 7fc825790640 1 -- 172.21.15.77:0/2272520651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc820068180 con 0x7fc82006feb0 2024-04-25T01:23:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.264+0000 7fc8077fe640 1 -- 172.21.15.77:0/2272520651 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc81401b070 con 0x7fc82006feb0 2024-04-25T01:23:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.264+0000 7fc8077fe640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc80c06be30 0x7fc80c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.264+0000 7fc8077fe640 1 -- 172.21.15.77:0/2272520651 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7fc81408a200 con 0x7fc82006feb0 2024-04-25T01:23:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.264+0000 7fc81effd640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc80c06be30 0x7fc80c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.264+0000 7fc81effd640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc80c06be30 0x7fc80c06e2f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc8200cdfb0 tx=0x7fc81000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:17.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.268+0000 7fc8077fe640 1 -- 172.21.15.77:0/2272520651 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc814058410 con 0x7fc82006feb0 2024-04-25T01:23:17.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.443+0000 7fc825790640 1 -- 172.21.15.77:0/2272520651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8200b7c00 con 0x7fc82006feb0 2024-04-25T01:23:17.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.445+0000 7fc8077fe640 1 -- 172.21.15.77:0/2272520651 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+17071 (secure 0 0 0) 0x7fc81405c710 con 0x7fc82006feb0 2024-04-25T01:23:17.425 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:17.425 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":39,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:23:06.238842+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.457887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.029883+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6837/2944320661":"2024-04-26T01:23:06.238820+0000","172.21.15.77:6836/2944320661":"2024-04-26T01:23:06.238820+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.446+0000 7fc825790640 1 -- 172.21.15.77:0/2272520651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc80c06be30 msgr2=0x7fc80c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.446+0000 7fc825790640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc80c06be30 0x7fc80c06e2f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc8200cdfb0 tx=0x7fc81000b040 comp rx=0 tx=0).stop 2024-04-25T01:23:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.446+0000 7fc825790640 1 -- 172.21.15.77:0/2272520651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc82006feb0 msgr2=0x7fc8201f3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.446+0000 7fc825790640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc82006feb0 0x7fc8201f3c40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc81400c500 tx=0x7fc81400c9c0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.447+0000 7fc825790640 1 -- 172.21.15.77:0/2272520651 shutdown_connections 2024-04-25T01:23:17.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.447+0000 7fc825790640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc80c06be30 0x7fc80c06e2f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.447+0000 7fc825790640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc82006feb0 0x7fc8201f3c40 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.447+0000 7fc825790640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8200ce500 0x7fc8200b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.447+0000 7fc825790640 1 --2- 172.21.15.77:0/2272520651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc82006fa50 0x7fc820156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.447+0000 7fc825790640 1 -- 172.21.15.77:0/2272520651 >> 172.21.15.77:0/2272520651 conn(0x7fc8200d1c80 msgr2=0x7fc8200afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:17.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.447+0000 7fc825790640 1 -- 172.21.15.77:0/2272520651 shutdown_connections 2024-04-25T01:23:17.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.447+0000 7fc825790640 1 -- 172.21.15.77:0/2272520651 wait complete. 2024-04-25T01:23:17.438 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T01:23:17.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.555+0000 7f17797b7640 1 Processor -- start 2024-04-25T01:23:17.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.555+0000 7f17797b7640 1 -- start start 2024-04-25T01:23:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.556+0000 7f17797b7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1774154ac0 0x7f177415b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.556+0000 7f17797b7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17740cf2a0 0x7f17740711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.556+0000 7f17797b7640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1774154ea0 con 0x7f1774071730 2024-04-25T01:23:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.556+0000 7f17797b7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f177406d540 con 0x7f1774154ac0 2024-04-25T01:23:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.556+0000 7f17797b7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f177406d790 con 0x7f17740cf2a0 2024-04-25T01:23:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.556+0000 7f1772ffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1774154ac0 0x7f177415b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.556+0000 7f17727fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17740cf2a0 0x7f17740711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.556+0000 7f1772ffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1774154ac0 0x7f177415b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:51892/0 (socket says 172.21.15.77:51892) 2024-04-25T01:23:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.556+0000 7f1772ffd640 1 -- 172.21.15.77:0/483449085 learned_addr learned my addr 172.21.15.77:0/483449085 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.556+0000 7f17737fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f1774071730 0x7f177406e550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:42660/0 (socket says 172.21.15.77:42660) 2024-04-25T01:23:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.557+0000 7f1772ffd640 1 -- 172.21.15.77:0/483449085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17740cf2a0 msgr2=0x7f17740711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.557+0000 7f1772ffd640 1 --2- 172.21.15.77:0/483449085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17740cf2a0 0x7f17740711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.557+0000 7f1772ffd640 1 -- 172.21.15.77:0/483449085 >> v1:172.21.15.121:6789/0 conn(0x7f1774071730 legacy=0x7f177406e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.557+0000 7f1772ffd640 1 -- 172.21.15.77:0/483449085 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177406da70 con 0x7f1774154ac0 2024-04-25T01:23:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.557+0000 7f17727fc640 1 --2- 172.21.15.77:0/483449085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17740cf2a0 0x7f17740711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:17.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.557+0000 7f1772ffd640 1 --2- 172.21.15.77:0/483449085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1774154ac0 0x7f177415b5f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1764009e50 tx=0x7f176402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39a038b9a558522d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:17.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.558+0000 7f1771ffb640 1 -- 172.21.15.77:0/483449085 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764036070 con 0x7f1774154ac0 2024-04-25T01:23:17.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.558+0000 7f1771ffb640 1 -- 172.21.15.77:0/483449085 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1764031600 con 0x7f1774154ac0 2024-04-25T01:23:17.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.558+0000 7f1771ffb640 1 -- 172.21.15.77:0/483449085 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17640319d0 con 0x7f1774154ac0 2024-04-25T01:23:17.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.558+0000 7f17797b7640 1 -- 172.21.15.77:0/483449085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1774154ac0 msgr2=0x7f177415b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.558+0000 7f17797b7640 1 --2- 172.21.15.77:0/483449085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1774154ac0 0x7f177415b5f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1764009e50 tx=0x7f176402a710 comp rx=0 tx=0).stop 2024-04-25T01:23:17.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.558+0000 7f17797b7640 1 -- 172.21.15.77:0/483449085 shutdown_connections 2024-04-25T01:23:17.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.558+0000 7f17797b7640 1 --2- 172.21.15.77:0/483449085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17740cf2a0 0x7f17740711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.558+0000 7f17797b7640 1 --2- 172.21.15.77:0/483449085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1774154ac0 0x7f177415b5f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.558+0000 7f17797b7640 1 -- 172.21.15.77:0/483449085 >> 172.21.15.77:0/483449085 conn(0x7f17740b1820 msgr2=0x7f17740b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:17.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.558+0000 7f17797b7640 1 -- 172.21.15.77:0/483449085 shutdown_connections 2024-04-25T01:23:17.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.558+0000 7f17797b7640 1 -- 172.21.15.77:0/483449085 wait complete. 2024-04-25T01:23:17.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.559+0000 7f17797b7640 1 Processor -- start 2024-04-25T01:23:17.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.560+0000 7f17797b7640 1 -- start start 2024-04-25T01:23:17.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.560+0000 7f17797b7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1774071730 0x7f1774159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.560+0000 7f17797b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17740cf2a0 0x7f1774159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.560+0000 7f17797b7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1774154ac0 0x7f1774145a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.560+0000 7f17797b7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f17740cf720 con 0x7f17740cf2a0 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.560+0000 7f17797b7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f177406e030 con 0x7f1774154ac0 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.560+0000 7f17797b7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f177406ec60 con 0x7f1774071730 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.560+0000 7f17727fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17740cf2a0 0x7f1774159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.560+0000 7f17727fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17740cf2a0 0x7f1774159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54460/0 (socket says 172.21.15.77:54460) 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.560+0000 7f17727fc640 1 -- 172.21.15.77:0/2368002465 learned_addr learned my addr 172.21.15.77:0/2368002465 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f17737fe640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1774154ac0 0x7f1774145a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f1772ffd640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1774071730 0x7f1774159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f17727fc640 1 -- 172.21.15.77:0/2368002465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1774071730 msgr2=0x7f1774159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f17727fc640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1774071730 0x7f1774159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f17727fc640 1 -- 172.21.15.77:0/2368002465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1774154ac0 msgr2=0x7f1774145a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f17727fc640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1774154ac0 0x7f1774145a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f17727fc640 1 -- 172.21.15.77:0/2368002465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1774141520 con 0x7f17740cf2a0 2024-04-25T01:23:17.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f17737fe640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1774154ac0 0x7f1774145a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:17.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f1772ffd640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1774071730 0x7f1774159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:17.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f17727fc640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17740cf2a0 0x7f1774159b40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f176800ebe0 tx=0x7f176800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:17.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f175b7fe640 1 -- 172.21.15.77:0/2368002465 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768017070 con 0x7f17740cf2a0 2024-04-25T01:23:17.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f175b7fe640 1 -- 172.21.15.77:0/2368002465 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f176800cc30 con 0x7f17740cf2a0 2024-04-25T01:23:17.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f175b7fe640 1 -- 172.21.15.77:0/2368002465 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768013470 con 0x7f17740cf2a0 2024-04-25T01:23:17.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f17797b7640 1 -- 172.21.15.77:0/2368002465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1774141800 con 0x7f17740cf2a0 2024-04-25T01:23:17.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.561+0000 7f17797b7640 1 -- 172.21.15.77:0/2368002465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1774141dc0 con 0x7f17740cf2a0 2024-04-25T01:23:17.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.562+0000 7f17797b7640 1 -- 172.21.15.77:0/2368002465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17740cc030 con 0x7f17740cf2a0 2024-04-25T01:23:17.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.562+0000 7f175b7fe640 1 -- 172.21.15.77:0/2368002465 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f176801a070 con 0x7f17740cf2a0 2024-04-25T01:23:17.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.563+0000 7f175b7fe640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f176006be30 0x7f176006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.563+0000 7f175b7fe640 1 -- 172.21.15.77:0/2368002465 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f176808aff0 con 0x7f17740cf2a0 2024-04-25T01:23:17.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.563+0000 7f1772ffd640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f176006be30 0x7f176006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.563+0000 7f1772ffd640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f176006be30 0x7f176006e2f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1764033000 tx=0x7f1764009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:17.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.567+0000 7f175b7fe640 1 -- 172.21.15.77:0/2368002465 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f17680597c0 con 0x7f17740cf2a0 2024-04-25T01:23:17.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.778+0000 7f17797b7640 1 -- 172.21.15.77:0/2368002465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1774068fb0 con 0x7f17740cf2a0 2024-04-25T01:23:17.760 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-04-25T01:23:17.779+0000 7f46b1a53640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:23:17.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.780+0000 7f175b7fe640 1 -- 172.21.15.77:0/2368002465 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v22) ==== 161+0+0 (secure 0 0 0) 0x7f17680590c0 con 0x7f17740cf2a0 2024-04-25T01:23:17.761 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T01:23:17.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 -- 172.21.15.77:0/2368002465 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f176006be30 msgr2=0x7f176006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f176006be30 0x7f176006e2f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1764033000 tx=0x7f1764009000 comp rx=0 tx=0).stop 2024-04-25T01:23:17.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 -- 172.21.15.77:0/2368002465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17740cf2a0 msgr2=0x7f1774159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17740cf2a0 0x7f1774159b40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f176800ebe0 tx=0x7f176800c520 comp rx=0 tx=0).stop 2024-04-25T01:23:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 -- 172.21.15.77:0/2368002465 shutdown_connections 2024-04-25T01:23:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f176006be30 0x7f176006e2f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17740cf2a0 0x7f1774159b40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1774154ac0 0x7f1774145a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 --2- 172.21.15.77:0/2368002465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1774071730 0x7f1774159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 -- 172.21.15.77:0/2368002465 >> 172.21.15.77:0/2368002465 conn(0x7f17740b1820 msgr2=0x7f17740726e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 -- 172.21.15.77:0/2368002465 shutdown_connections 2024-04-25T01:23:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.783+0000 7f17797b7640 1 -- 172.21.15.77:0/2368002465 wait complete. 2024-04-25T01:23:17.765 INFO:tasks.ceph.mds.c.smithi077.stderr: -980> 2024-04-25T01:23:11.697+0000 7f2de6706640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:23:17.766 INFO:tasks.ceph.mds.c.smithi077.stderr: -690> 2024-04-25T01:23:11.823+0000 7f2de26fe640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x10000000000 2024-04-25T01:23:17.776 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T01:23:17.791 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-04-25T01:23:17.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.894+0000 7f7730a66640 1 Processor -- start 2024-04-25T01:23:17.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.894+0000 7f7730a66640 1 -- start start 2024-04-25T01:23:17.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.894+0000 7f7730a66640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c0d27b0 0x7f772c0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.894+0000 7f7730a66640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f772c0cea40 con 0x7f772c0b9a90 2024-04-25T01:23:17.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.894+0000 7f7730a66640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f772c0cece0 con 0x7f772c0d23d0 2024-04-25T01:23:17.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.894+0000 7f7730a66640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f772c0cefc0 con 0x7f772c0d27b0 2024-04-25T01:23:17.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.894+0000 7f7723fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f772c0b9a90 0x7f772c0b6c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44240/0 (socket says 172.21.15.77:44240) 2024-04-25T01:23:17.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.894+0000 7f7723fff640 1 -- 172.21.15.77:0/2356375209 learned_addr learned my addr 172.21.15.77:0/2356375209 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:17.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3309546447 0 0) 0x7f772c0cea40 con 0x7f772c0b9a90 2024-04-25T01:23:17.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7718003520 con 0x7f772c0b9a90 2024-04-25T01:23:17.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f772ad9d640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c0d27b0 0x7f772c0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 797028312 0 0) 0x7f7718003520 con 0x7f772c0b9a90 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c0d27b0 msgr2=0x7f772c0d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c0d27b0 0x7f772c0d4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 >> v1:172.21.15.121:6789/0 conn(0x7f772c0d23d0 legacy=0x7f772c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f772c0cf2a0 con 0x7f772c0b9a90 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f7714001670 con 0x7f772c0b9a90 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 >> v1:172.21.15.77:6789/0 conn(0x7f772c0b9a90 legacy=0x7f772c0b6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c0d27b0 0x7f77180039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7718003f60 0x7f7718004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7718004d40 0x7f77180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f772c0cefc0 con 0x7f7718004d40 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f771800bb30 con 0x7f7718003f60 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f771800bd30 con 0x7f772c0d27b0 2024-04-25T01:23:17.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f772ad9d640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7718004d40 0x7f77180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f7723fff640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7718003f60 0x7f7718004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f772a59c640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c0d27b0 0x7f77180039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f772ad9d640 1 -- 172.21.15.77:0/2356375209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c0d27b0 msgr2=0x7f77180039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f772ad9d640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c0d27b0 0x7f77180039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f772ad9d640 1 -- 172.21.15.77:0/2356375209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7718003f60 msgr2=0x7f7718004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f772ad9d640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7718003f60 0x7f7718004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.895+0000 7f772ad9d640 1 -- 172.21.15.77:0/2356375209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7718003520 con 0x7f7718004d40 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.896+0000 7f772ad9d640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7718004d40 0x7f77180051c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7724009fc0 tx=0x7f7718005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fe4adb1bd1652b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.896+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7724009c10 con 0x7f7718004d40 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.896+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7724036070 con 0x7f7718004d40 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.896+0000 7f7729d9b640 1 -- 172.21.15.77:0/2356375209 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7724031880 con 0x7f7718004d40 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.897+0000 7f7730a66640 1 -- 172.21.15.77:0/2356375209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7718004d40 msgr2=0x7f77180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.897+0000 7f7730a66640 1 --2- 172.21.15.77:0/2356375209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7718004d40 0x7f77180051c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7724009fc0 tx=0x7f7718005830 comp rx=0 tx=0).stop 2024-04-25T01:23:17.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.897+0000 7f772ad9d640 1 -- 172.21.15.77:0/2356375209 reap_dead start 2024-04-25T01:23:17.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.897+0000 7f7730a66640 1 -- 172.21.15.77:0/2356375209 shutdown_connections 2024-04-25T01:23:17.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.897+0000 7f7730a66640 1 -- 172.21.15.77:0/2356375209 >> 172.21.15.77:0/2356375209 conn(0x7f772c0d3e30 msgr2=0x7f772c0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:17.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.897+0000 7f7730a66640 1 -- 172.21.15.77:0/2356375209 shutdown_connections 2024-04-25T01:23:17.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.897+0000 7f7730a66640 1 -- 172.21.15.77:0/2356375209 wait complete. 2024-04-25T01:23:17.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.898+0000 7f7730a66640 1 Processor -- start 2024-04-25T01:23:17.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.898+0000 7f7730a66640 1 -- start start 2024-04-25T01:23:17.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7730a66640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7718004d40 0x7f772c15d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7730a66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f772c1e85b0 0x7f772c0c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7730a66640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c1e8a30 0x7f772c1f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7730a66640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f772c0cfd80 con 0x7f772c1e85b0 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7730a66640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f772c0cf860 con 0x7f7718004d40 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7730a66640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f772c0d4920 con 0x7f772c1e8a30 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7723fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f772c1e85b0 0x7f772c0c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7723fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f772c1e85b0 0x7f772c0c9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54496/0 (socket says 172.21.15.77:54496) 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7723fff640 1 -- 172.21.15.77:0/3119822454 learned_addr learned my addr 172.21.15.77:0/3119822454 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7723fff640 1 -- 172.21.15.77:0/3119822454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c1e8a30 msgr2=0x7f772c1f3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7723fff640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c1e8a30 0x7f772c1f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7723fff640 1 -- 172.21.15.77:0/3119822454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7718004d40 msgr2=0x7f772c15d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7723fff640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7718004d40 0x7f772c15d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7723fff640 1 -- 172.21.15.77:0/3119822454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f772c1e7510 con 0x7f772c1e85b0 2024-04-25T01:23:17.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7723fff640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f772c1e85b0 0x7f772c0c9790 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f77140072c0 tx=0x7f7714007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7722ffd640 1 -- 172.21.15.77:0/3119822454 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771400f8e0 con 0x7f772c1e85b0 2024-04-25T01:23:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7722ffd640 1 -- 172.21.15.77:0/3119822454 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7714018070 con 0x7f772c1e85b0 2024-04-25T01:23:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7730a66640 1 -- 172.21.15.77:0/3119822454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f772c1e77f0 con 0x7f772c1e85b0 2024-04-25T01:23:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7722ffd640 1 -- 172.21.15.77:0/3119822454 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771400fce0 con 0x7f772c1e85b0 2024-04-25T01:23:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.899+0000 7f7730a66640 1 -- 172.21.15.77:0/3119822454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f772c1e7db0 con 0x7f772c1e85b0 2024-04-25T01:23:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.900+0000 7f7730a66640 1 -- 172.21.15.77:0/3119822454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f772c068fb0 con 0x7f772c1e85b0 2024-04-25T01:23:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.901+0000 7f7722ffd640 1 -- 172.21.15.77:0/3119822454 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f771401d460 con 0x7f772c1e85b0 2024-04-25T01:23:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.901+0000 7f7722ffd640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f770c06be30 0x7f770c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.901+0000 7f772a59c640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f770c06be30 0x7f770c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.901+0000 7f7722ffd640 1 -- 172.21.15.77:0/3119822454 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6273+0+0 (secure 0 0 0) 0x7f771408aff0 con 0x7f772c1e85b0 2024-04-25T01:23:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.902+0000 7f772a59c640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f770c06be30 0x7f770c06e2f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f771c001750 tx=0x7f771c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:17.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:17.906+0000 7f7722ffd640 1 -- 172.21.15.77:0/3119822454 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f77140597f0 con 0x7f772c1e85b0 2024-04-25T01:23:18.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.112+0000 7f7730a66640 1 -- 172.21.15.77:0/3119822454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f772c0ca770 con 0x7f772c1e85b0 2024-04-25T01:23:18.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.782+0000 7f7722ffd640 1 -- 172.21.15.77:0/3119822454 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v23) ==== 104+0+0 (secure 0 0 0) 0x7f77140590f0 con 0x7f772c1e85b0 2024-04-25T01:23:18.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 -- 172.21.15.77:0/3119822454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f770c06be30 msgr2=0x7f770c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:18.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f770c06be30 0x7f770c06e2f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f771c001750 tx=0x7f771c009040 comp rx=0 tx=0).stop 2024-04-25T01:23:18.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 -- 172.21.15.77:0/3119822454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f772c1e85b0 msgr2=0x7f772c0c9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:18.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f772c1e85b0 0x7f772c0c9790 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f77140072c0 tx=0x7f7714007780 comp rx=0 tx=0).stop 2024-04-25T01:23:18.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 -- 172.21.15.77:0/3119822454 shutdown_connections 2024-04-25T01:23:18.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f770c06be30 0x7f770c06e2f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:18.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f772c1e85b0 0x7f772c0c9790 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:18.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f772c1e8a30 0x7f772c1f3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:18.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 --2- 172.21.15.77:0/3119822454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7718004d40 0x7f772c15d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:18.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 -- 172.21.15.77:0/3119822454 >> 172.21.15.77:0/3119822454 conn(0x7f772c0d3e30 msgr2=0x7f772c06c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:18.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 -- 172.21.15.77:0/3119822454 shutdown_connections 2024-04-25T01:23:18.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.785+0000 7f7730a66640 1 -- 172.21.15.77:0/3119822454 wait complete. 2024-04-25T01:23:18.779 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T01:23:18.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.942+0000 7fcdf72f5640 1 Processor -- start 2024-04-25T01:23:18.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.943+0000 7fcdf72f5640 1 -- start start 2024-04-25T01:23:18.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.943+0000 7fcdf72f5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcdf015e4c0 0x7fcdf00d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.943+0000 7fcdf72f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00d4a10 0x7fcdf00b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.943+0000 7fcdf72f5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf00d1e90 con 0x7fcdf00d2d40 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.943+0000 7fcdf72f5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcdf00b54a0 con 0x7fcdf015e4c0 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.943+0000 7fcdf72f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf00d2130 con 0x7fcdf00d4a10 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcdf5892640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcdf00d2d40 0x7fcdf00d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44244/0 (socket says 172.21.15.77:44244) 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcdf5892640 1 -- 172.21.15.77:0/581142631 learned_addr learned my addr 172.21.15.77:0/581142631 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcdf5091640 1 --2- 172.21.15.77:0/581142631 >> v2:172.21.15.121:3300/0 conn(0x7fcdf015e4c0 0x7fcdf00d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcdf4890640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00d4a10 0x7fcdf00b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1852177475 0 0) 0x7fcdf00d1e90 con 0x7fcdf00d2d40 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcddc003520 con 0x7fcdf00d2d40 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 858243479 0 0) 0x7fcddc003520 con 0x7fcdf00d2d40 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00d4a10 msgr2=0x7fcdf00b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00d4a10 0x7fcdf00b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:18.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 >> v2:172.21.15.121:3300/0 conn(0x7fcdf015e4c0 msgr2=0x7fcdf00d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 --2- 172.21.15.77:0/581142631 >> v2:172.21.15.121:3300/0 conn(0x7fcdf015e4c0 0x7fcdf00d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf00b5780 con 0x7fcdf00d2d40 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fcde80019f0 con 0x7fcdf00d2d40 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcdf5091640 1 --2- 172.21.15.77:0/581142631 >> v2:172.21.15.121:3300/0 conn(0x7fcdf015e4c0 0x7fcdf00d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 >> v1:172.21.15.77:6789/0 conn(0x7fcdf00d2d40 legacy=0x7fcdf00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcddc0039b0 0x7fcddc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcddc004800 0x7fcddc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00d4a10 0x7fcddc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf00b54a0 con 0x7fcddc004800 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf00d2130 con 0x7fcddc0039b0 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcddc00b9c0 con 0x7fcdf00d4a10 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcdf4890640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcddc004800 0x7fcddc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcdf5091640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcddc0039b0 0x7fcddc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.944+0000 7fcdf5892640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00d4a10 0x7fcddc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.945+0000 7fcdf4890640 1 -- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00d4a10 msgr2=0x7fcddc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:18.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.945+0000 7fcdf4890640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00d4a10 0x7fcddc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.945+0000 7fcdf4890640 1 -- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcddc0039b0 msgr2=0x7fcddc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.945+0000 7fcdf4890640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcddc0039b0 0x7fcddc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.945+0000 7fcdf4890640 1 -- 172.21.15.77:0/581142631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcddc003520 con 0x7fcddc004800 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.945+0000 7fcdf5091640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcddc0039b0 0x7fcddc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.945+0000 7fcdf4890640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcddc004800 0x7fcddc004c80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009ac0 tx=0x7fcddc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc74d1a3810eb662 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.945+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec036070 con 0x7fcddc004800 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.945+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdec031400 con 0x7fcddc004800 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.945+0000 7fcde3fff640 1 -- 172.21.15.77:0/581142631 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec009690 con 0x7fcddc004800 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.946+0000 7fcdf72f5640 1 -- 172.21.15.77:0/581142631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcddc004800 msgr2=0x7fcddc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.946+0000 7fcdf72f5640 1 --2- 172.21.15.77:0/581142631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcddc004800 0x7fcddc004c80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009ac0 tx=0x7fcddc005830 comp rx=0 tx=0).stop 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.946+0000 7fcdf5892640 1 -- 172.21.15.77:0/581142631 reap_dead start 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.946+0000 7fcdf72f5640 1 -- 172.21.15.77:0/581142631 shutdown_connections 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.946+0000 7fcdf72f5640 1 -- 172.21.15.77:0/581142631 >> 172.21.15.77:0/581142631 conn(0x7fcdf00d3ef0 msgr2=0x7fcdf00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:18.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.946+0000 7fcdf72f5640 1 -- 172.21.15.77:0/581142631 shutdown_connections 2024-04-25T01:23:18.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.946+0000 7fcdf72f5640 1 -- 172.21.15.77:0/581142631 wait complete. 2024-04-25T01:23:18.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.947+0000 7fcdf72f5640 1 Processor -- start 2024-04-25T01:23:18.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.947+0000 7fcdf72f5640 1 -- start start 2024-04-25T01:23:18.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf72f5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcddc004800 0x7fcdf00cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:18.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf72f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf014de60 0x7fcdf014d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:18.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf72f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00c82f0 0x7fcdf00c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:18.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf72f5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf015ea20 con 0x7fcdf014de60 2024-04-25T01:23:18.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf72f5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf015e8a0 con 0x7fcddc004800 2024-04-25T01:23:18.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf72f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf00d1010 con 0x7fcdf00c82f0 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf4890640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf014de60 0x7fcdf014d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf4890640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf014de60 0x7fcdf014d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54514/0 (socket says 172.21.15.77:54514) 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf4890640 1 -- 172.21.15.77:0/1854795006 learned_addr learned my addr 172.21.15.77:0/1854795006 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf5892640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00c82f0 0x7fcdf00c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf5091640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcddc004800 0x7fcdf00cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf4890640 1 -- 172.21.15.77:0/1854795006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00c82f0 msgr2=0x7fcdf00c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf4890640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00c82f0 0x7fcdf00c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf4890640 1 -- 172.21.15.77:0/1854795006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcddc004800 msgr2=0x7fcdf00cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf4890640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcddc004800 0x7fcdf00cdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf4890640 1 -- 172.21.15.77:0/1854795006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf00b7c00 con 0x7fcdf014de60 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf5091640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcddc004800 0x7fcdf00cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf5892640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00c82f0 0x7fcdf00c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.948+0000 7fcdf4890640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf014de60 0x7fcdf014d640 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fcdec034040 tx=0x7fcdec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.949+0000 7fcde1ffb640 1 -- 172.21.15.77:0/1854795006 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec041020 con 0x7fcdf014de60 2024-04-25T01:23:18.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.949+0000 7fcde1ffb640 1 -- 172.21.15.77:0/1854795006 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdec03c470 con 0x7fcdf014de60 2024-04-25T01:23:18.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.949+0000 7fcdf72f5640 1 -- 172.21.15.77:0/1854795006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf00b7e70 con 0x7fcdf014de60 2024-04-25T01:23:18.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.949+0000 7fcde1ffb640 1 -- 172.21.15.77:0/1854795006 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec036040 con 0x7fcdf014de60 2024-04-25T01:23:18.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.949+0000 7fcdf72f5640 1 -- 172.21.15.77:0/1854795006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf00b8430 con 0x7fcdf014de60 2024-04-25T01:23:18.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.950+0000 7fcdf72f5640 1 -- 172.21.15.77:0/1854795006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdf0068fb0 con 0x7fcdf014de60 2024-04-25T01:23:18.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.950+0000 7fcde1ffb640 1 -- 172.21.15.77:0/1854795006 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcdec044020 con 0x7fcdf014de60 2024-04-25T01:23:18.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.951+0000 7fcde1ffb640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcdd406be60 0x7fcdd406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:18.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.951+0000 7fcdf5091640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcdd406be60 0x7fcdd406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:18.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.951+0000 7fcde1ffb640 1 -- 172.21.15.77:0/1854795006 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6273+0+0 (secure 0 0 0) 0x7fcdec0aaef0 con 0x7fcdf014de60 2024-04-25T01:23:18.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.951+0000 7fcdf5091640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcdd406be60 0x7fcdd406e320 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009ef0 tx=0x7fcde4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:18.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:18.955+0000 7fcde1ffb640 1 -- 172.21.15.77:0/1854795006 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcdec0796d0 con 0x7fcdf014de60 2024-04-25T01:23:19.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.127+0000 7fcdf72f5640 1 -- 172.21.15.77:0/1854795006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcdf00d0a30 con 0x7fcdf014de60 2024-04-25T01:23:19.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.787+0000 7fcde1ffb640 1 -- 172.21.15.77:0/1854795006 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v41) ==== 181+0+0 (secure 0 0 0) 0x7fcdec078fd0 con 0x7fcdf014de60 2024-04-25T01:23:19.768 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-04-25T01:23:19.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.790+0000 7fcdf72f5640 1 -- 172.21.15.77:0/1854795006 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcdd406be60 msgr2=0x7fcdd406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.790+0000 7fcdf72f5640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcdd406be60 0x7fcdd406e320 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009ef0 tx=0x7fcde4008040 comp rx=0 tx=0).stop 2024-04-25T01:23:19.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.790+0000 7fcdf72f5640 1 -- 172.21.15.77:0/1854795006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf014de60 msgr2=0x7fcdf014d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.790+0000 7fcdf72f5640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf014de60 0x7fcdf014d640 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fcdec034040 tx=0x7fcdec004730 comp rx=0 tx=0).stop 2024-04-25T01:23:19.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.790+0000 7fcdf72f5640 1 -- 172.21.15.77:0/1854795006 shutdown_connections 2024-04-25T01:23:19.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.790+0000 7fcdf72f5640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcdd406be60 0x7fcdd406e320 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.790+0000 7fcdf72f5640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf014de60 0x7fcdf014d640 secure :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fcdec034040 tx=0x7fcdec004730 comp rx=0 tx=0).stop 2024-04-25T01:23:19.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.790+0000 7fcdf72f5640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdf00c82f0 0x7fcdf00c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.790+0000 7fcdf72f5640 1 --2- 172.21.15.77:0/1854795006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcddc004800 0x7fcdf00cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.790+0000 7fcdf72f5640 1 -- 172.21.15.77:0/1854795006 >> 172.21.15.77:0/1854795006 conn(0x7fcdf00d3ef0 msgr2=0x7fcdf00b3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:19.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.791+0000 7fcdf72f5640 1 -- 172.21.15.77:0/1854795006 shutdown_connections 2024-04-25T01:23:19.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.791+0000 7fcdf72f5640 1 -- 172.21.15.77:0/1854795006 wait complete. 2024-04-25T01:23:19.785 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T01:23:19.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.911+0000 7fbea53af640 1 Processor -- start 2024-04-25T01:23:19.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.911+0000 7fbea53af640 1 -- start start 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbea53af640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbea00d4a10 0x7fbea00cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbea53af640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea00d1e90 con 0x7fbea00b3ee0 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbea53af640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea00b54a0 con 0x7fbea015e4c0 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbea53af640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fbea00d2130 con 0x7fbea00d4a10 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9f7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbea00b3ee0 0x7fbea00d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44254/0 (socket says 172.21.15.77:44254) 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9f7fe640 1 -- 172.21.15.77:0/1008810276 learned_addr learned my addr 172.21.15.77:0/1008810276 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9e7fc640 1 --2- 172.21.15.77:0/1008810276 >> v2:172.21.15.121:3300/0 conn(0x7fbea00d4a10 0x7fbea00cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806590316 0 0) 0x7fbea00d1e90 con 0x7fbea00b3ee0 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe88003520 con 0x7fbea00b3ee0 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2037044515 0 0) 0x7fbe88003520 con 0x7fbea00b3ee0 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 >> v2:172.21.15.121:3300/0 conn(0x7fbea00d4a10 msgr2=0x7fbea00cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 --2- 172.21.15.77:0/1008810276 >> v2:172.21.15.121:3300/0 conn(0x7fbea00d4a10 0x7fbea00cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 >> v1:172.21.15.121:6789/0 conn(0x7fbea015e4c0 legacy=0x7fbea00d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea00b5780 con 0x7fbea00b3ee0 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fbe980019f0 con 0x7fbea00b3ee0 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 >> v1:172.21.15.77:6789/0 conn(0x7fbea00b3ee0 legacy=0x7fbea00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 --2- 172.21.15.77:0/1008810276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe880039b0 0x7fbe88003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 --2- 172.21.15.77:0/1008810276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe880043b0 0x7fbe88004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 --2- 172.21.15.77:0/1008810276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe88005120 0x7fbe8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbea00d2130 con 0x7fbe880039b0 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8800c060 con 0x7fbe880043b0 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.912+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbe8800c260 con 0x7fbe88005120 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.913+0000 7fbe9effd640 1 --2- 172.21.15.77:0/1008810276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe880039b0 0x7fbe88003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.913+0000 7fbe9e7fc640 1 --2- 172.21.15.77:0/1008810276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe880043b0 0x7fbe88004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.913+0000 7fbe9f7fe640 1 --2- 172.21.15.77:0/1008810276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe88005120 0x7fbe8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.913+0000 7fbe9effd640 1 -- 172.21.15.77:0/1008810276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe88005120 msgr2=0x7fbe8800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.913+0000 7fbe9effd640 1 --2- 172.21.15.77:0/1008810276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe88005120 0x7fbe8800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.913+0000 7fbe9effd640 1 -- 172.21.15.77:0/1008810276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe880043b0 msgr2=0x7fbe88004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:19.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.913+0000 7fbe9effd640 1 --2- 172.21.15.77:0/1008810276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe880043b0 0x7fbe88004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.913+0000 7fbe9effd640 1 -- 172.21.15.77:0/1008810276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe88003520 con 0x7fbe880039b0 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.913+0000 7fbe9f7fe640 1 -- 172.21.15.77:0/1008810276 reap_dead start 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.913+0000 7fbe9effd640 1 --2- 172.21.15.77:0/1008810276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe880039b0 0x7fbe88003d90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbe90001860 tx=0x7fbe88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f23a933ffe64fb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.914+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe90037070 con 0x7fbe880039b0 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.914+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe9002ac30 con 0x7fbe880039b0 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.914+0000 7fbe9dffb640 1 -- 172.21.15.77:0/1008810276 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe90001430 con 0x7fbe880039b0 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.914+0000 7fbea53af640 1 -- 172.21.15.77:0/1008810276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe880039b0 msgr2=0x7fbe88003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.914+0000 7fbea53af640 1 --2- 172.21.15.77:0/1008810276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe880039b0 0x7fbe88003d90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbe90001860 tx=0x7fbe88005830 comp rx=0 tx=0).stop 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.914+0000 7fbea53af640 1 -- 172.21.15.77:0/1008810276 shutdown_connections 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.914+0000 7fbea53af640 1 --2- 172.21.15.77:0/1008810276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe880039b0 0x7fbe88003d90 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.914+0000 7fbea53af640 1 -- 172.21.15.77:0/1008810276 >> 172.21.15.77:0/1008810276 conn(0x7fbea00d3ef0 msgr2=0x7fbea00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.914+0000 7fbea53af640 1 -- 172.21.15.77:0/1008810276 shutdown_connections 2024-04-25T01:23:19.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.914+0000 7fbea53af640 1 -- 172.21.15.77:0/1008810276 wait complete. 2024-04-25T01:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.915+0000 7fbea53af640 1 Processor -- start 2024-04-25T01:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbea53af640 1 -- start start 2024-04-25T01:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbea53af640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe880039b0 0x7fbea00b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbea53af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea00cdb20 0x7fbea0161590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbea53af640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea0161ad0 0x7fbea015ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbea53af640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbea015ea20 con 0x7fbea00cdb20 2024-04-25T01:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbea53af640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbea015e8a0 con 0x7fbea0161ad0 2024-04-25T01:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbea53af640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbea00b4370 con 0x7fbe880039b0 2024-04-25T01:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbe9e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea00cdb20 0x7fbea0161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbe9e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea00cdb20 0x7fbea0161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54528/0 (socket says 172.21.15.77:54528) 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbe9e7fc640 1 -- 172.21.15.77:0/1408899197 learned_addr learned my addr 172.21.15.77:0/1408899197 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbe9f7fe640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea0161ad0 0x7fbea015ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.916+0000 7fbe9effd640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe880039b0 0x7fbea00b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.917+0000 7fbe9e7fc640 1 -- 172.21.15.77:0/1408899197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe880039b0 msgr2=0x7fbea00b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.917+0000 7fbe9e7fc640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe880039b0 0x7fbea00b1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.917+0000 7fbe9e7fc640 1 -- 172.21.15.77:0/1408899197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea0161ad0 msgr2=0x7fbea015ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.917+0000 7fbe9e7fc640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea0161ad0 0x7fbea015ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.917+0000 7fbe9e7fc640 1 -- 172.21.15.77:0/1408899197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea00b0f20 con 0x7fbea00cdb20 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.917+0000 7fbe9e7fc640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea00cdb20 0x7fbea0161590 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fbe94011980 tx=0x7fbe94011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.917+0000 7fbe877fe640 1 -- 172.21.15.77:0/1408899197 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe94019070 con 0x7fbea00cdb20 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.917+0000 7fbe877fe640 1 -- 172.21.15.77:0/1408899197 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe94004240 con 0x7fbea00cdb20 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.917+0000 7fbe877fe640 1 -- 172.21.15.77:0/1408899197 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe9400f8e0 con 0x7fbea00cdb20 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.917+0000 7fbea53af640 1 -- 172.21.15.77:0/1408899197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea00b1200 con 0x7fbea00cdb20 2024-04-25T01:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.917+0000 7fbea53af640 1 -- 172.21.15.77:0/1408899197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea00b17c0 con 0x7fbea00cdb20 2024-04-25T01:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.918+0000 7fbea53af640 1 -- 172.21.15.77:0/1408899197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea0068fb0 con 0x7fbea00cdb20 2024-04-25T01:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.918+0000 7fbe877fe640 1 -- 172.21.15.77:0/1408899197 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbe9401c070 con 0x7fbea00cdb20 2024-04-25T01:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.919+0000 7fbe877fe640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be30 0x7fbe8c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.919+0000 7fbe877fe640 1 -- 172.21.15.77:0/1408899197 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 5835+0+0 (secure 0 0 0) 0x7fbe9408b310 con 0x7fbea00cdb20 2024-04-25T01:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.919+0000 7fbe9effd640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be30 0x7fbe8c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.919+0000 7fbe9effd640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be30 0x7fbe8c06e2f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbe90034000 tx=0x7fbe90009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:19.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:19.923+0000 7fbe877fe640 1 -- 172.21.15.77:0/1408899197 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbe94059550 con 0x7fbea00cdb20 2024-04-25T01:23:20.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.095+0000 7fbea53af640 1 -- 172.21.15.77:0/1408899197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbea00b9b40 con 0x7fbea00cdb20 2024-04-25T01:23:20.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.792+0000 7fbe877fe640 1 -- 172.21.15.77:0/1408899197 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v42) ==== 169+0+0 (secure 0 0 0) 0x7fbe9405d850 con 0x7fbea00cdb20 2024-04-25T01:23:20.773 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-04-25T01:23:20.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.795+0000 7fbea53af640 1 -- 172.21.15.77:0/1408899197 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be30 msgr2=0x7fbe8c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.795+0000 7fbea53af640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be30 0x7fbe8c06e2f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbe90034000 tx=0x7fbe90009030 comp rx=0 tx=0).stop 2024-04-25T01:23:20.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.795+0000 7fbea53af640 1 -- 172.21.15.77:0/1408899197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea00cdb20 msgr2=0x7fbea0161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.795+0000 7fbea53af640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea00cdb20 0x7fbea0161590 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fbe94011980 tx=0x7fbe94011e40 comp rx=0 tx=0).stop 2024-04-25T01:23:20.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.796+0000 7fbea53af640 1 -- 172.21.15.77:0/1408899197 shutdown_connections 2024-04-25T01:23:20.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.796+0000 7fbea53af640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fbe8c06be30 0x7fbe8c06e2f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.796+0000 7fbea53af640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea00cdb20 0x7fbea0161590 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.796+0000 7fbea53af640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea0161ad0 0x7fbea015ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.796+0000 7fbea53af640 1 --2- 172.21.15.77:0/1408899197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe880039b0 0x7fbea00b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.796+0000 7fbea53af640 1 -- 172.21.15.77:0/1408899197 >> 172.21.15.77:0/1408899197 conn(0x7fbea00d3ef0 msgr2=0x7fbea00ceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.796+0000 7fbea53af640 1 -- 172.21.15.77:0/1408899197 shutdown_connections 2024-04-25T01:23:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.796+0000 7fbea53af640 1 -- 172.21.15.77:0/1408899197 wait complete. 2024-04-25T01:23:20.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T01:23:20.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.912+0000 7f6c223f9640 1 Processor -- start 2024-04-25T01:23:20.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.912+0000 7f6c223f9640 1 -- start start 2024-04-25T01:23:20.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.912+0000 7f6c223f9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c1c0d23d0 0x7f6c1c0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.912+0000 7f6c223f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1c0b9a90 0x7f6c1c0d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.912+0000 7f6c223f9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c1c0cea40 con 0x7f6c1c0d27b0 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.912+0000 7f6c223f9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6c1c0cece0 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.912+0000 7f6c223f9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1c0cefc0 con 0x7f6c1c0b9a90 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.913+0000 7f6c1bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c1c0d23d0 0x7f6c1c0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.913+0000 7f6c1b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6c1c0d27b0 0x7f6c1c0b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44264/0 (socket says 172.21.15.77:44264) 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.913+0000 7f6c1b7fe640 1 -- 172.21.15.77:0/406963135 learned_addr learned my addr 172.21.15.77:0/406963135 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.913+0000 7f6c1bfff640 1 -- 172.21.15.77:0/406963135 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1c0b9a90 msgr2=0x7f6c1c0d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.913+0000 7f6c1bfff640 1 --2- 172.21.15.77:0/406963135 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1c0b9a90 0x7f6c1c0d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.913+0000 7f6c1bfff640 1 -- 172.21.15.77:0/406963135 >> v1:172.21.15.77:6789/0 conn(0x7f6c1c0d27b0 legacy=0x7f6c1c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.913+0000 7f6c1affd640 1 -- 172.21.15.77:0/406963135 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1122803306 0 0) 0x7f6c1c0cea40 con 0x7f6c1c0d27b0 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.913+0000 7f6c1bfff640 1 -- 172.21.15.77:0/406963135 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1c0cf2a0 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.913+0000 7f6c1bfff640 1 --2- 172.21.15.77:0/406963135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c1c0d23d0 0x7f6c1c0d4920 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c0097a0 tx=0x7f6c0c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7e76a1f144eb70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:20.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.914+0000 7f6c1affd640 1 -- 172.21.15.77:0/406963135 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c037070 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.914+0000 7f6c1affd640 1 -- 172.21.15.77:0/406963135 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c0c032440 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.914+0000 7f6c1affd640 1 -- 172.21.15.77:0/406963135 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c032810 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.914+0000 7f6c223f9640 1 -- 172.21.15.77:0/406963135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c1c0d23d0 msgr2=0x7f6c1c0d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.914+0000 7f6c223f9640 1 --2- 172.21.15.77:0/406963135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c1c0d23d0 0x7f6c1c0d4920 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c0097a0 tx=0x7f6c0c02a210 comp rx=0 tx=0).stop 2024-04-25T01:23:20.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.914+0000 7f6c223f9640 1 -- 172.21.15.77:0/406963135 shutdown_connections 2024-04-25T01:23:20.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.914+0000 7f6c223f9640 1 --2- 172.21.15.77:0/406963135 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1c0b9a90 0x7f6c1c0d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.914+0000 7f6c223f9640 1 --2- 172.21.15.77:0/406963135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c1c0d23d0 0x7f6c1c0d4920 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.914+0000 7f6c223f9640 1 -- 172.21.15.77:0/406963135 >> 172.21.15.77:0/406963135 conn(0x7f6c1c0d3e30 msgr2=0x7f6c1c0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:20.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.914+0000 7f6c223f9640 1 -- 172.21.15.77:0/406963135 shutdown_connections 2024-04-25T01:23:20.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.914+0000 7f6c223f9640 1 -- 172.21.15.77:0/406963135 wait complete. 2024-04-25T01:23:20.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.915+0000 7f6c223f9640 1 Processor -- start 2024-04-25T01:23:20.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.916+0000 7f6c223f9640 1 -- start start 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.916+0000 7f6c223f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1c0b9a90 0x7f6c1c14e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.916+0000 7f6c223f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c1c0d23d0 0x7f6c1c0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.916+0000 7f6c223f9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c1c0d27b0 0x7f6c1c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.916+0000 7f6c223f9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c0cfd80 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.916+0000 7f6c223f9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c0cf860 con 0x7f6c1c0d27b0 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.916+0000 7f6c223f9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1c0b6c80 con 0x7f6c1c0b9a90 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.916+0000 7f6c1b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c1c0d23d0 0x7f6c1c0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.916+0000 7f6c1b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c1c0d23d0 0x7f6c1c0cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54548/0 (socket says 172.21.15.77:54548) 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.916+0000 7f6c1b7fe640 1 -- 172.21.15.77:0/2645625040 learned_addr learned my addr 172.21.15.77:0/2645625040 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c1bfff640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1c0b9a90 0x7f6c1c14e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c20996640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c1c0d27b0 0x7f6c1c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c1b7fe640 1 -- 172.21.15.77:0/2645625040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1c0b9a90 msgr2=0x7f6c1c14e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c1b7fe640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1c0b9a90 0x7f6c1c14e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c1b7fe640 1 -- 172.21.15.77:0/2645625040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c1c0d27b0 msgr2=0x7f6c1c0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c1b7fe640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c1c0d27b0 0x7f6c1c0cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c1b7fe640 1 -- 172.21.15.77:0/2645625040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1c15bf20 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c20996640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c1c0d27b0 0x7f6c1c0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:20.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c1b7fe640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c1c0d23d0 0x7f6c1c0cc1f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000c460 tx=0x7f6c1000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:20.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c18ff9640 1 -- 172.21.15.77:0/2645625040 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10018070 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c18ff9640 1 -- 172.21.15.77:0/2645625040 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c10007500 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c223f9640 1 -- 172.21.15.77:0/2645625040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1c15c200 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c18ff9640 1 -- 172.21.15.77:0/2645625040 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10004a20 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.917+0000 7f6c223f9640 1 -- 172.21.15.77:0/2645625040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1c15c7c0 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.918+0000 7f6c223f9640 1 -- 172.21.15.77:0/2645625040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c1c068fb0 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.919+0000 7f6c18ff9640 1 -- 172.21.15.77:0/2645625040 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6c1001b070 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.919+0000 7f6c18ff9640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c0806be30 0x7f6c0806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.919+0000 7f6c18ff9640 1 -- 172.21.15.77:0/2645625040 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7f6c10089f60 con 0x7f6c1c0d23d0 2024-04-25T01:23:20.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.919+0000 7f6c1bfff640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c0806be30 0x7f6c0806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.919+0000 7f6c1bfff640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c0806be30 0x7f6c0806e2f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c034000 tx=0x7f6c0c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:20.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:20.923+0000 7f6c18ff9640 1 -- 172.21.15.77:0/2645625040 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6c10058ac0 con 0x7f6c1c0d23d0 2024-04-25T01:23:21.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.098+0000 7f6c223f9640 1 -- 172.21.15.77:0/2645625040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f6c1c08df50 con 0x7f6c1c0d23d0 2024-04-25T01:23:21.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.101+0000 7f6c18ff9640 1 -- 172.21.15.77:0/2645625040 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f6c100583c0 con 0x7f6c1c0d23d0 2024-04-25T01:23:21.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 -- 172.21.15.77:0/2645625040 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c0806be30 msgr2=0x7f6c0806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c0806be30 0x7f6c0806e2f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c034000 tx=0x7f6c0c02e000 comp rx=0 tx=0).stop 2024-04-25T01:23:21.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 -- 172.21.15.77:0/2645625040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c1c0d23d0 msgr2=0x7f6c1c0cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c1c0d23d0 0x7f6c1c0cc1f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000c460 tx=0x7f6c1000c920 comp rx=0 tx=0).stop 2024-04-25T01:23:21.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 -- 172.21.15.77:0/2645625040 shutdown_connections 2024-04-25T01:23:21.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6c0806be30 0x7f6c0806e2f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c1c0d23d0 0x7f6c1c0cc1f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c1c0d27b0 0x7f6c1c0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 --2- 172.21.15.77:0/2645625040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1c0b9a90 0x7f6c1c14e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 -- 172.21.15.77:0/2645625040 >> 172.21.15.77:0/2645625040 conn(0x7f6c1c0d3e30 msgr2=0x7f6c1c161310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:21.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 -- 172.21.15.77:0/2645625040 shutdown_connections 2024-04-25T01:23:21.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.104+0000 7f6c223f9640 1 -- 172.21.15.77:0/2645625040 wait complete. 2024-04-25T01:23:21.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_backtrace_repair' 2024-04-25T01:23:21.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.212+0000 7f3153276640 1 Processor -- start 2024-04-25T01:23:21.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.212+0000 7f3153276640 1 -- start start 2024-04-25T01:23:21.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.213+0000 7f3153276640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f314c143940 0x7f314c06e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.213+0000 7f3153276640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c143d20 0x7f314c06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.213+0000 7f3153276640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f314c15ecf0 0x7f314c15f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.213+0000 7f3153276640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f314c070e60 con 0x7f314c15ecf0 2024-04-25T01:23:21.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.213+0000 7f3153276640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f314c071140 con 0x7f314c143940 2024-04-25T01:23:21.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.213+0000 7f3153276640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f314c071420 con 0x7f314c143d20 2024-04-25T01:23:21.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.213+0000 7f3151813640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f314c15ecf0 0x7f314c15f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.213+0000 7f3151813640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f314c15ecf0 0x7f314c15f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54560/0 (socket says 172.21.15.77:54560) 2024-04-25T01:23:21.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.213+0000 7f3151813640 1 -- 172.21.15.77:0/1569336107 learned_addr learned my addr 172.21.15.77:0/1569336107 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:21.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.213+0000 7f3151813640 1 -- 172.21.15.77:0/1569336107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c143d20 msgr2=0x7f314c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:21.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.213+0000 7f3151813640 1 --2- 172.21.15.77:0/1569336107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c143d20 0x7f314c06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.214+0000 7f3151813640 1 -- 172.21.15.77:0/1569336107 >> v2:172.21.15.121:3300/0 conn(0x7f314c143940 msgr2=0x7f314c06e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:21.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.214+0000 7f3151012640 1 --2- 172.21.15.77:0/1569336107 >> v2:172.21.15.121:3300/0 conn(0x7f314c143940 0x7f314c06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.214+0000 7f3151813640 1 --2- 172.21.15.77:0/1569336107 >> v2:172.21.15.121:3300/0 conn(0x7f314c143940 0x7f314c06e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.214+0000 7f3151813640 1 -- 172.21.15.77:0/1569336107 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314c071700 con 0x7f314c15ecf0 2024-04-25T01:23:21.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.214+0000 7f3151813640 1 --2- 172.21.15.77:0/1569336107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f314c15ecf0 0x7f314c15f180 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f314c06c150 tx=0x7f314402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47951695fbd9d5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:21.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.214+0000 7f3143fff640 1 -- 172.21.15.77:0/1569336107 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3144037070 con 0x7f314c15ecf0 2024-04-25T01:23:21.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.214+0000 7f3143fff640 1 -- 172.21.15.77:0/1569336107 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3144032440 con 0x7f314c15ecf0 2024-04-25T01:23:21.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.214+0000 7f3143fff640 1 -- 172.21.15.77:0/1569336107 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3144032810 con 0x7f314c15ecf0 2024-04-25T01:23:21.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.215+0000 7f3153276640 1 -- 172.21.15.77:0/1569336107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f314c15ecf0 msgr2=0x7f314c15f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.215+0000 7f3153276640 1 --2- 172.21.15.77:0/1569336107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f314c15ecf0 0x7f314c15f180 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f314c06c150 tx=0x7f314402a1e0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.215+0000 7f3153276640 1 -- 172.21.15.77:0/1569336107 shutdown_connections 2024-04-25T01:23:21.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.215+0000 7f3153276640 1 --2- 172.21.15.77:0/1569336107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f314c15ecf0 0x7f314c15f180 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.215+0000 7f3153276640 1 --2- 172.21.15.77:0/1569336107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c143d20 0x7f314c06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.215+0000 7f3153276640 1 --2- 172.21.15.77:0/1569336107 >> v2:172.21.15.121:3300/0 conn(0x7f314c143940 0x7f314c06e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.215+0000 7f3153276640 1 -- 172.21.15.77:0/1569336107 >> 172.21.15.77:0/1569336107 conn(0x7f314c0b1820 msgr2=0x7f314c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:21.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.215+0000 7f3153276640 1 -- 172.21.15.77:0/1569336107 shutdown_connections 2024-04-25T01:23:21.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.215+0000 7f3153276640 1 -- 172.21.15.77:0/1569336107 wait complete. 2024-04-25T01:23:21.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.216+0000 7f3153276640 1 Processor -- start 2024-04-25T01:23:21.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.216+0000 7f3153276640 1 -- start start 2024-04-25T01:23:21.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3153276640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f314c143940 0x7f314c13ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3153276640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f314c143d20 0x7f314c1f4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3153276640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c15ecf0 0x7f314c0cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3153276640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f314c0b3a00 con 0x7f314c143d20 2024-04-25T01:23:21.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3153276640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f314c0b3880 con 0x7f314c143940 2024-04-25T01:23:21.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3153276640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f314c0b3b80 con 0x7f314c15ecf0 2024-04-25T01:23:21.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3150811640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f314c143d20 0x7f314c1f4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3150811640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f314c143d20 0x7f314c1f4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54566/0 (socket says 172.21.15.77:54566) 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3150811640 1 -- 172.21.15.77:0/3066508530 learned_addr learned my addr 172.21.15.77:0/3066508530 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3150811640 1 -- 172.21.15.77:0/3066508530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c15ecf0 msgr2=0x7f314c0cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3150811640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c15ecf0 0x7f314c0cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3150811640 1 -- 172.21.15.77:0/3066508530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f314c143940 msgr2=0x7f314c13ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3150811640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f314c143940 0x7f314c13ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3150811640 1 -- 172.21.15.77:0/3066508530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314c0af2c0 con 0x7f314c143d20 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3150811640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f314c143d20 0x7f314c1f4f30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f313c00b4a0 tx=0x7f313c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3141ffb640 1 -- 172.21.15.77:0/3066508530 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313c014070 con 0x7f314c143d20 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.217+0000 7f3141ffb640 1 -- 172.21.15.77:0/3066508530 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f313c010610 con 0x7f314c143d20 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.218+0000 7f3153276640 1 -- 172.21.15.77:0/3066508530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314c13f1a0 con 0x7f314c143d20 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.218+0000 7f3141ffb640 1 -- 172.21.15.77:0/3066508530 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313c010a00 con 0x7f314c143d20 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.218+0000 7f3153276640 1 -- 172.21.15.77:0/3066508530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f314c0aefe0 con 0x7f314c143d20 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.219+0000 7f3153276640 1 -- 172.21.15.77:0/3066508530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f314c0cc0b0 con 0x7f314c143d20 2024-04-25T01:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.219+0000 7f3141ffb640 1 -- 172.21.15.77:0/3066508530 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f313c017070 con 0x7f314c143d20 2024-04-25T01:23:21.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.220+0000 7f3141ffb640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f313006be30 0x7f313006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.220+0000 7f3151012640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f313006be30 0x7f313006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.220+0000 7f3141ffb640 1 -- 172.21.15.77:0/3066508530 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7f313c08a6d0 con 0x7f314c143d20 2024-04-25T01:23:21.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.220+0000 7f3151012640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f313006be30 0x7f313006e2f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3148007e60 tx=0x7f3148007390 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:21.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.224+0000 7f3141ffb640 1 -- 172.21.15.77:0/3066508530 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f314c0cc0b0 con 0x7f314c143d20 2024-04-25T01:23:21.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.463+0000 7f3153276640 1 -- 172.21.15.77:0/3066508530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_backtrace_repair"]} v 0) -- 0x7f314c068fb0 con 0x7f314c143d20 2024-04-25T01:23:21.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.793+0000 7f3141ffb640 1 -- 172.21.15.77:0/3066508530 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_backtrace_repair"]}]=0 v114) ==== 148+0+0 (secure 0 0 0) 0x7f313c059230 con 0x7f314c143d20 2024-04-25T01:23:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 -- 172.21.15.77:0/3066508530 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f313006be30 msgr2=0x7f313006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f313006be30 0x7f313006e2f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3148007e60 tx=0x7f3148007390 comp rx=0 tx=0).stop 2024-04-25T01:23:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 -- 172.21.15.77:0/3066508530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f314c143d20 msgr2=0x7f314c1f4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f314c143d20 0x7f314c1f4f30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f313c00b4a0 tx=0x7f313c00b960 comp rx=0 tx=0).stop 2024-04-25T01:23:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 -- 172.21.15.77:0/3066508530 shutdown_connections 2024-04-25T01:23:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f313006be30 0x7f313006e2f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f314c143d20 0x7f314c1f4f30 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c15ecf0 0x7f314c0cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 --2- 172.21.15.77:0/3066508530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f314c143940 0x7f314c13ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 -- 172.21.15.77:0/3066508530 >> 172.21.15.77:0/3066508530 conn(0x7f314c0b1820 msgr2=0x7f314c06e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 -- 172.21.15.77:0/3066508530 shutdown_connections 2024-04-25T01:23:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.795+0000 7f3153276640 1 -- 172.21.15.77:0/3066508530 wait complete. 2024-04-25T01:23:21.787 INFO:tasks.cephfs_test_runner:MDS should verify/fix backtrace on fetch dirfrag ... ok 2024-04-25T01:23:21.787 INFO:tasks.cephfs_test_runner:Starting test: test_mds_readonly (tasks.cephfs.test_auto_repair.TestMDSAutoRepair) test if MDS behave correct when it's readonly 2024-04-25T01:23:21.788 INFO:tasks.cephfs_test_runner:test_mds_readonly (tasks.cephfs.test_auto_repair.TestMDSAutoRepair) 2024-04-25T01:23:21.788 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:21.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.904+0000 7f1856742640 1 Processor -- start 2024-04-25T01:23:21.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.904+0000 7f1856742640 1 -- start start 2024-04-25T01:23:21.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.905+0000 7f1856742640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1850143d20 0x7f185015ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.905+0000 7f1856742640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1850070e60 con 0x7f1850143940 2024-04-25T01:23:21.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.905+0000 7f1856742640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1850071140 con 0x7f185006ed60 2024-04-25T01:23:21.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.905+0000 7f1856742640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f1850071420 con 0x7f1850143d20 2024-04-25T01:23:21.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.905+0000 7f184f7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1850143d20 0x7f185015ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.905+0000 7f184ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1850143940 0x7f185006e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44270/0 (socket says 172.21.15.77:44270) 2024-04-25T01:23:21.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.905+0000 7f184ffff640 1 -- 172.21.15.77:0/124923653 learned_addr learned my addr 172.21.15.77:0/124923653 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:21.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.905+0000 7f184f7fe640 1 -- 172.21.15.77:0/124923653 >> v1:172.21.15.121:6789/0 conn(0x7f185006ed60 legacy=0x7f185015f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:21.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.905+0000 7f184f7fe640 1 -- 172.21.15.77:0/124923653 >> v1:172.21.15.77:6789/0 conn(0x7f1850143940 legacy=0x7f185006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.905+0000 7f184f7fe640 1 -- 172.21.15.77:0/124923653 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1850071700 con 0x7f1850143d20 2024-04-25T01:23:21.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.905+0000 7f184effd640 1 -- 172.21.15.77:0/124923653 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 8132167 0 0) 0x7f1850070e60 con 0x7f1850143940 2024-04-25T01:23:21.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.906+0000 7f184f7fe640 1 --2- 172.21.15.77:0/124923653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1850143d20 0x7f185015ecf0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f18440097d0 tx=0x7f184402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3bd22e63da57b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:21.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.906+0000 7f184effd640 1 -- 172.21.15.77:0/124923653 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844037070 con 0x7f1850143d20 2024-04-25T01:23:21.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.906+0000 7f184effd640 1 -- 172.21.15.77:0/124923653 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1844032440 con 0x7f1850143d20 2024-04-25T01:23:21.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.906+0000 7f184effd640 1 -- 172.21.15.77:0/124923653 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844032810 con 0x7f1850143d20 2024-04-25T01:23:21.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.906+0000 7f1856742640 1 -- 172.21.15.77:0/124923653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1850143d20 msgr2=0x7f185015ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.906+0000 7f1856742640 1 --2- 172.21.15.77:0/124923653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1850143d20 0x7f185015ecf0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f18440097d0 tx=0x7f184402a210 comp rx=0 tx=0).stop 2024-04-25T01:23:21.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.906+0000 7f1856742640 1 -- 172.21.15.77:0/124923653 shutdown_connections 2024-04-25T01:23:21.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.906+0000 7f1856742640 1 --2- 172.21.15.77:0/124923653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1850143d20 0x7f185015ecf0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.906+0000 7f1856742640 1 -- 172.21.15.77:0/124923653 >> 172.21.15.77:0/124923653 conn(0x7f18500b1820 msgr2=0x7f18500b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:21.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.906+0000 7f1856742640 1 -- 172.21.15.77:0/124923653 shutdown_connections 2024-04-25T01:23:21.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.906+0000 7f1856742640 1 -- 172.21.15.77:0/124923653 wait complete. 2024-04-25T01:23:21.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.908+0000 7f1856742640 1 Processor -- start 2024-04-25T01:23:21.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.908+0000 7f1856742640 1 -- start start 2024-04-25T01:23:21.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.908+0000 7f1856742640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f185006ed60 0x7f185015c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.908+0000 7f1856742640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1850143940 0x7f1850140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.908+0000 7f1856742640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1850143d20 0x7f185013a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.908+0000 7f1856742640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1850165d30 con 0x7f1850143d20 2024-04-25T01:23:21.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.908+0000 7f1856742640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1850072e80 con 0x7f185006ed60 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.908+0000 7f1856742640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1850165eb0 con 0x7f1850143940 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f1854cdf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1850143d20 0x7f185013a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f1854cdf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1850143d20 0x7f185013a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54596/0 (socket says 172.21.15.77:54596) 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f1854cdf640 1 -- 172.21.15.77:0/2833046778 learned_addr learned my addr 172.21.15.77:0/2833046778 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f1854cdf640 1 -- 172.21.15.77:0/2833046778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1850143940 msgr2=0x7f1850140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f184f7fe640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1850143940 0x7f1850140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f184ffff640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f185006ed60 0x7f185015c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f1854cdf640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1850143940 0x7f1850140520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f1854cdf640 1 -- 172.21.15.77:0/2833046778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f185006ed60 msgr2=0x7f185015c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f1854cdf640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f185006ed60 0x7f185015c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f1854cdf640 1 -- 172.21.15.77:0/2833046778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18500ccff0 con 0x7f1850143d20 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f1854cdf640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1850143d20 0x7f185013a270 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f184800c780 tx=0x7f184800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f184cff9640 1 -- 172.21.15.77:0/2833046778 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848017070 con 0x7f1850143d20 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f184cff9640 1 -- 172.21.15.77:0/2833046778 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1848003da0 con 0x7f1850143d20 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f1856742640 1 -- 172.21.15.77:0/2833046778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18500cd2d0 con 0x7f1850143d20 2024-04-25T01:23:21.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f184cff9640 1 -- 172.21.15.77:0/2833046778 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18480052b0 con 0x7f1850143d20 2024-04-25T01:23:21.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.909+0000 7f1856742640 1 -- 172.21.15.77:0/2833046778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18500cd890 con 0x7f1850143d20 2024-04-25T01:23:21.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.910+0000 7f1856742640 1 -- 172.21.15.77:0/2833046778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1850068fb0 con 0x7f1850143d20 2024-04-25T01:23:21.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.911+0000 7f184cff9640 1 -- 172.21.15.77:0/2833046778 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f184801a070 con 0x7f1850143d20 2024-04-25T01:23:21.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.911+0000 7f184cff9640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f183c06be30 0x7f183c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.911+0000 7f184cff9640 1 -- 172.21.15.77:0/2833046778 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7f184808a370 con 0x7f1850143d20 2024-04-25T01:23:21.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.911+0000 7f184ffff640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f183c06be30 0x7f183c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.912+0000 7f184ffff640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f183c06be30 0x7f183c06e2f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1840007fd0 tx=0x7f1840009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:21.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:21.915+0000 7f184cff9640 1 -- 172.21.15.77:0/2833046778 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1848058e40 con 0x7f1850143d20 2024-04-25T01:23:22.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.089+0000 7f1856742640 1 -- 172.21.15.77:0/2833046778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f185006d150 con 0x7f1850143d20 2024-04-25T01:23:22.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.090+0000 7f184cff9640 1 -- 172.21.15.77:0/2833046778 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+13423 (secure 0 0 0) 0x7f1848058740 con 0x7f1850143d20 2024-04-25T01:23:22.071 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:22.071 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":42,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:23:20.789061+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6835/3139470828":"2024-04-26T01:23:17.780231+0000","172.21.15.77:6834/3139470828":"2024-04-26T01:23:17.780231+0000","172.21.15.77:6837/2944320661":"2024-04-26T01:23:06.238820+0000","172.21.15.77:6836/2944320661":"2024-04-26T01:23:06.238820+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:22.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 -- 172.21.15.77:0/2833046778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f183c06be30 msgr2=0x7f183c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f183c06be30 0x7f183c06e2f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1840007fd0 tx=0x7f1840009040 comp rx=0 tx=0).stop 2024-04-25T01:23:22.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 -- 172.21.15.77:0/2833046778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1850143d20 msgr2=0x7f185013a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1850143d20 0x7f185013a270 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f184800c780 tx=0x7f184800cc40 comp rx=0 tx=0).stop 2024-04-25T01:23:22.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 -- 172.21.15.77:0/2833046778 shutdown_connections 2024-04-25T01:23:22.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f183c06be30 0x7f183c06e2f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1850143d20 0x7f185013a270 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1850143940 0x7f1850140520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 --2- 172.21.15.77:0/2833046778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f185006ed60 0x7f185015c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 -- 172.21.15.77:0/2833046778 >> 172.21.15.77:0/2833046778 conn(0x7f18500b1820 msgr2=0x7f18500cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:22.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 -- 172.21.15.77:0/2833046778 shutdown_connections 2024-04-25T01:23:22.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.092+0000 7f1856742640 1 -- 172.21.15.77:0/2833046778 wait complete. 2024-04-25T01:23:22.084 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:23:22.084 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:23:22.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.205+0000 7f6846653640 1 Processor -- start 2024-04-25T01:23:22.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.205+0000 7f6846653640 1 -- start start 2024-04-25T01:23:22.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6846653640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f68400722e0 0x7f684006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6846653640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f68400cf4d0 0x7f68401600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6846653640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f684006e410 con 0x7f684015fc40 2024-04-25T01:23:22.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6846653640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6840160aa0 con 0x7f68400cf4d0 2024-04-25T01:23:22.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6846653640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f684006da40 con 0x7f68400722e0 2024-04-25T01:23:22.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6844bf0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f68400cf4d0 0x7f68401600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6844bf0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f68400cf4d0 0x7f68401600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54612/0 (socket says 172.21.15.77:54612) 2024-04-25T01:23:22.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6844bf0640 1 -- 172.21.15.77:0/2294413550 learned_addr learned my addr 172.21.15.77:0/2294413550 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:22.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f683ffff640 1 --2- 172.21.15.77:0/2294413550 >> v2:172.21.15.121:3300/0 conn(0x7f68400722e0 0x7f684006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6844bf0640 1 -- 172.21.15.77:0/2294413550 >> v2:172.21.15.121:3300/0 conn(0x7f68400722e0 msgr2=0x7f684006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:22.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6844bf0640 1 --2- 172.21.15.77:0/2294413550 >> v2:172.21.15.121:3300/0 conn(0x7f68400722e0 0x7f684006ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6844bf0640 1 -- 172.21.15.77:0/2294413550 >> v1:172.21.15.121:6789/0 conn(0x7f684015fc40 legacy=0x7f68400cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.206+0000 7f6844bf0640 1 -- 172.21.15.77:0/2294413550 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684006d110 con 0x7f68400cf4d0 2024-04-25T01:23:22.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.207+0000 7f6844bf0640 1 --2- 172.21.15.77:0/2294413550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68400cf4d0 0x7f68401600c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f68400727b0 tx=0x7f683802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a9ee9a5839770a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:22.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.207+0000 7f683effd640 1 -- 172.21.15.77:0/2294413550 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838036070 con 0x7f68400cf4d0 2024-04-25T01:23:22.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.207+0000 7f683effd640 1 -- 172.21.15.77:0/2294413550 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6838031600 con 0x7f68400cf4d0 2024-04-25T01:23:22.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.207+0000 7f683effd640 1 -- 172.21.15.77:0/2294413550 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68380319d0 con 0x7f68400cf4d0 2024-04-25T01:23:22.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.207+0000 7f6846653640 1 -- 172.21.15.77:0/2294413550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68400cf4d0 msgr2=0x7f68401600c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.207+0000 7f6846653640 1 --2- 172.21.15.77:0/2294413550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68400cf4d0 0x7f68401600c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f68400727b0 tx=0x7f683802a6a0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.208+0000 7f6846653640 1 -- 172.21.15.77:0/2294413550 shutdown_connections 2024-04-25T01:23:22.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.208+0000 7f6846653640 1 --2- 172.21.15.77:0/2294413550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68400cf4d0 0x7f68401600c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.208+0000 7f6846653640 1 --2- 172.21.15.77:0/2294413550 >> v2:172.21.15.121:3300/0 conn(0x7f68400722e0 0x7f684006ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.208+0000 7f6846653640 1 -- 172.21.15.77:0/2294413550 >> 172.21.15.77:0/2294413550 conn(0x7f684006b720 msgr2=0x7f68400b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:22.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.208+0000 7f6846653640 1 -- 172.21.15.77:0/2294413550 shutdown_connections 2024-04-25T01:23:22.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.208+0000 7f6846653640 1 -- 172.21.15.77:0/2294413550 wait complete. 2024-04-25T01:23:22.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.209+0000 7f6846653640 1 Processor -- start 2024-04-25T01:23:22.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.209+0000 7f6846653640 1 -- start start 2024-04-25T01:23:22.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f6846653640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68400722e0 0x7f68401432e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f6846653640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68400cf4d0 0x7f6840155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f6846653640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f684015fc40 0x7f6840132400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f6846653640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f684006d6d0 con 0x7f68400cf4d0 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f6846653640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f684006cec0 con 0x7f684015fc40 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f6846653640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6840070b50 con 0x7f68400722e0 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68400cf4d0 0x7f6840155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68400cf4d0 0x7f6840155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54628/0 (socket says 172.21.15.77:54628) 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683f7fe640 1 -- 172.21.15.77:0/778076591 learned_addr learned my addr 172.21.15.77:0/778076591 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683f7fe640 1 -- 172.21.15.77:0/778076591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68400722e0 msgr2=0x7f68401432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683f7fe640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68400722e0 0x7f68401432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683f7fe640 1 -- 172.21.15.77:0/778076591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f684015fc40 msgr2=0x7f6840132400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f6844bf0640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f684015fc40 0x7f6840132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683f7fe640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f684015fc40 0x7f6840132400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683f7fe640 1 -- 172.21.15.77:0/778076591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6840133290 con 0x7f68400cf4d0 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683f7fe640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68400cf4d0 0x7f6840155b70 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f683000c3c0 tx=0x7f683000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:22.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683cff9640 1 -- 172.21.15.77:0/778076591 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68300052d0 con 0x7f68400cf4d0 2024-04-25T01:23:22.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683cff9640 1 -- 172.21.15.77:0/778076591 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6830005490 con 0x7f68400cf4d0 2024-04-25T01:23:22.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f683cff9640 1 -- 172.21.15.77:0/778076591 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6830007260 con 0x7f68400cf4d0 2024-04-25T01:23:22.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f6846653640 1 -- 172.21.15.77:0/778076591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68401571a0 con 0x7f68400cf4d0 2024-04-25T01:23:22.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.210+0000 7f6846653640 1 -- 172.21.15.77:0/778076591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6840142dc0 con 0x7f68400cf4d0 2024-04-25T01:23:22.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.211+0000 7f6846653640 1 -- 172.21.15.77:0/778076591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6840142fc0 con 0x7f68400cf4d0 2024-04-25T01:23:22.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.212+0000 7f683cff9640 1 -- 172.21.15.77:0/778076591 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6830007400 con 0x7f68400cf4d0 2024-04-25T01:23:22.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.212+0000 7f683cff9640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f682006be30 0x7f682006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.212+0000 7f683cff9640 1 -- 172.21.15.77:0/778076591 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7f683008a280 con 0x7f68400cf4d0 2024-04-25T01:23:22.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.212+0000 7f683ffff640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f682006be30 0x7f682006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.213+0000 7f683ffff640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f682006be30 0x7f682006e2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f68280098f0 tx=0x7f6828008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:22.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.216+0000 7f683cff9640 1 -- 172.21.15.77:0/778076591 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6840142fc0 con 0x7f68400cf4d0 2024-04-25T01:23:22.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.388+0000 7f6846653640 1 -- 172.21.15.77:0/778076591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6840068fb0 con 0x7f68400cf4d0 2024-04-25T01:23:22.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.388+0000 7f683cff9640 1 -- 172.21.15.77:0/778076591 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f6830058cc0 con 0x7f68400cf4d0 2024-04-25T01:23:22.370 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:23:22.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 -- 172.21.15.77:0/778076591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f682006be30 msgr2=0x7f682006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f682006be30 0x7f682006e2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f68280098f0 tx=0x7f6828008040 comp rx=0 tx=0).stop 2024-04-25T01:23:22.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 -- 172.21.15.77:0/778076591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68400cf4d0 msgr2=0x7f6840155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68400cf4d0 0x7f6840155b70 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f683000c3c0 tx=0x7f683000c880 comp rx=0 tx=0).stop 2024-04-25T01:23:22.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 -- 172.21.15.77:0/778076591 shutdown_connections 2024-04-25T01:23:22.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f682006be30 0x7f682006e2f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68400cf4d0 0x7f6840155b70 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f684015fc40 0x7f6840132400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 --2- 172.21.15.77:0/778076591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68400722e0 0x7f68401432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 -- 172.21.15.77:0/778076591 >> 172.21.15.77:0/778076591 conn(0x7f684006b720 msgr2=0x7f68401441a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:22.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 -- 172.21.15.77:0/778076591 shutdown_connections 2024-04-25T01:23:22.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.390+0000 7f6846653640 1 -- 172.21.15.77:0/778076591 wait complete. 2024-04-25T01:23:22.382 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_mds_readonly' 2024-04-25T01:23:22.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.497+0000 7fe04039e640 1 Processor -- start 2024-04-25T01:23:22.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.497+0000 7fe04039e640 1 -- start start 2024-04-25T01:23:22.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe04039e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe038143940 0x7fe03806e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe04039e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe03815f400 0x7fe03806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe04039e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe038070e60 con 0x7fe038143d20 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe04039e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fe038071140 con 0x7fe03815f400 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe04039e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe038071420 con 0x7fe038143940 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe03e93b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe03815f400 0x7fe03806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe03d939640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe038143d20 0x7fe03815ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44286/0 (socket says 172.21.15.77:44286) 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe03d939640 1 -- 172.21.15.77:0/947765958 learned_addr learned my addr 172.21.15.77:0/947765958 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe03e13a640 1 --2- 172.21.15.77:0/947765958 >> v2:172.21.15.121:3300/0 conn(0x7fe038143940 0x7fe03806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe03e93b640 1 -- 172.21.15.77:0/947765958 >> v2:172.21.15.121:3300/0 conn(0x7fe038143940 msgr2=0x7fe03806e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe03e93b640 1 --2- 172.21.15.77:0/947765958 >> v2:172.21.15.121:3300/0 conn(0x7fe038143940 0x7fe03806e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe03e93b640 1 -- 172.21.15.77:0/947765958 >> v1:172.21.15.77:6789/0 conn(0x7fe038143d20 legacy=0x7fe03815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe03e93b640 1 -- 172.21.15.77:0/947765958 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe038071700 con 0x7fe03815f400 2024-04-25T01:23:22.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.498+0000 7fe03d138640 1 -- 172.21.15.77:0/947765958 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 447150438 0 0) 0x7fe038070e60 con 0x7fe038143d20 2024-04-25T01:23:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.499+0000 7fe03e93b640 1 --2- 172.21.15.77:0/947765958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03815f400 0x7fe03806eb90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe0300097a0 tx=0x7fe03002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51cfe4c6082acdc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.499+0000 7fe03d138640 1 -- 172.21.15.77:0/947765958 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe030037070 con 0x7fe03815f400 2024-04-25T01:23:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.499+0000 7fe03d138640 1 -- 172.21.15.77:0/947765958 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe030032440 con 0x7fe03815f400 2024-04-25T01:23:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.499+0000 7fe03d138640 1 -- 172.21.15.77:0/947765958 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe030032810 con 0x7fe03815f400 2024-04-25T01:23:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.500+0000 7fe04039e640 1 -- 172.21.15.77:0/947765958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03815f400 msgr2=0x7fe03806eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.500+0000 7fe04039e640 1 --2- 172.21.15.77:0/947765958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03815f400 0x7fe03806eb90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe0300097a0 tx=0x7fe03002a1e0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.500+0000 7fe04039e640 1 -- 172.21.15.77:0/947765958 shutdown_connections 2024-04-25T01:23:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.500+0000 7fe04039e640 1 --2- 172.21.15.77:0/947765958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03815f400 0x7fe03806eb90 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.500+0000 7fe04039e640 1 --2- 172.21.15.77:0/947765958 >> v2:172.21.15.121:3300/0 conn(0x7fe038143940 0x7fe03806e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.500+0000 7fe04039e640 1 -- 172.21.15.77:0/947765958 >> 172.21.15.77:0/947765958 conn(0x7fe0380b1820 msgr2=0x7fe0380b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:22.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.500+0000 7fe04039e640 1 -- 172.21.15.77:0/947765958 shutdown_connections 2024-04-25T01:23:22.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.500+0000 7fe04039e640 1 -- 172.21.15.77:0/947765958 wait complete. 2024-04-25T01:23:22.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.501+0000 7fe04039e640 1 Processor -- start 2024-04-25T01:23:22.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.501+0000 7fe04039e640 1 -- start start 2024-04-25T01:23:22.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe04039e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe038143940 0x7fe03815e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe04039e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe038143d20 0x7fe0381589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe04039e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe03815f400 0x7fe03815a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe04039e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe03806d8d0 con 0x7fe038143d20 2024-04-25T01:23:22.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe04039e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe03806d750 con 0x7fe038143940 2024-04-25T01:23:22.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe04039e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe03806da50 con 0x7fe03815f400 2024-04-25T01:23:22.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe03d939640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe038143d20 0x7fe0381589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe03d939640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe038143d20 0x7fe0381589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54632/0 (socket says 172.21.15.77:54632) 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe03d939640 1 -- 172.21.15.77:0/3802843673 learned_addr learned my addr 172.21.15.77:0/3802843673 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe03e13a640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe038143940 0x7fe03815e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe03e93b640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe03815f400 0x7fe03815a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe03d939640 1 -- 172.21.15.77:0/3802843673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe03815f400 msgr2=0x7fe03815a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe03d939640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe03815f400 0x7fe03815a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe03d939640 1 -- 172.21.15.77:0/3802843673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe038143940 msgr2=0x7fe03815e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe03d939640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe038143940 0x7fe03815e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe03d939640 1 -- 172.21.15.77:0/3802843673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03815a700 con 0x7fe038143d20 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe03d939640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe038143d20 0x7fe0381589b0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe02800c4d0 tx=0x7fe02800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe02effd640 1 -- 172.21.15.77:0/3802843673 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe028018070 con 0x7fe038143d20 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.502+0000 7fe02effd640 1 -- 172.21.15.77:0/3802843673 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe028007500 con 0x7fe038143d20 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.503+0000 7fe04039e640 1 -- 172.21.15.77:0/3802843673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe038159120 con 0x7fe038143d20 2024-04-25T01:23:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.503+0000 7fe02effd640 1 -- 172.21.15.77:0/3802843673 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe028004a20 con 0x7fe038143d20 2024-04-25T01:23:22.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.503+0000 7fe04039e640 1 -- 172.21.15.77:0/3802843673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0381596e0 con 0x7fe038143d20 2024-04-25T01:23:22.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.504+0000 7fe04039e640 1 -- 172.21.15.77:0/3802843673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe038068fb0 con 0x7fe038143d20 2024-04-25T01:23:22.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.504+0000 7fe02effd640 1 -- 172.21.15.77:0/3802843673 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe02801b070 con 0x7fe038143d20 2024-04-25T01:23:22.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.504+0000 7fe02effd640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe01c06be30 0x7fe01c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.504+0000 7fe02effd640 1 -- 172.21.15.77:0/3802843673 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7fe02808a010 con 0x7fe038143d20 2024-04-25T01:23:22.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.504+0000 7fe03e13a640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe01c06be30 0x7fe01c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.505+0000 7fe03e13a640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe01c06be30 0x7fe01c06e2f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe0340074a0 tx=0x7fe03400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:22.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.508+0000 7fe02effd640 1 -- 172.21.15.77:0/3802843673 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe028058b70 con 0x7fe038143d20 2024-04-25T01:23:22.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.746+0000 7fe04039e640 1 -- 172.21.15.77:0/3802843673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_mds_readonly"]} v 0) -- 0x7fe03806d150 con 0x7fe038143d20 2024-04-25T01:23:22.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.794+0000 7fe02effd640 1 -- 172.21.15.77:0/3802843673 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_mds_readonly"]}]=0 v115) ==== 147+0+0 (secure 0 0 0) 0x7fe028058470 con 0x7fe038143d20 2024-04-25T01:23:22.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.796+0000 7fe04039e640 1 -- 172.21.15.77:0/3802843673 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe01c06be30 msgr2=0x7fe01c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.796+0000 7fe04039e640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe01c06be30 0x7fe01c06e2f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe0340074a0 tx=0x7fe03400b040 comp rx=0 tx=0).stop 2024-04-25T01:23:22.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.796+0000 7fe04039e640 1 -- 172.21.15.77:0/3802843673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe038143d20 msgr2=0x7fe0381589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.796+0000 7fe04039e640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe038143d20 0x7fe0381589b0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe02800c4d0 tx=0x7fe02800c990 comp rx=0 tx=0).stop 2024-04-25T01:23:22.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.796+0000 7fe04039e640 1 -- 172.21.15.77:0/3802843673 shutdown_connections 2024-04-25T01:23:22.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.796+0000 7fe04039e640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe01c06be30 0x7fe01c06e2f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.796+0000 7fe04039e640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe038143d20 0x7fe0381589b0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.796+0000 7fe04039e640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe03815f400 0x7fe03815a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.797+0000 7fe04039e640 1 --2- 172.21.15.77:0/3802843673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe038143940 0x7fe03815e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.797+0000 7fe04039e640 1 -- 172.21.15.77:0/3802843673 >> 172.21.15.77:0/3802843673 conn(0x7fe0380b1820 msgr2=0x7fe038072430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:22.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.797+0000 7fe04039e640 1 -- 172.21.15.77:0/3802843673 shutdown_connections 2024-04-25T01:23:22.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.797+0000 7fe04039e640 1 -- 172.21.15.77:0/3802843673 wait complete. 2024-04-25T01:23:22.788 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T01:23:22.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.907+0000 7f3d96458640 1 Processor -- start 2024-04-25T01:23:22.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.907+0000 7f3d96458640 1 -- start start 2024-04-25T01:23:22.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.908+0000 7f3d96458640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d900720c0 0x7f3d900b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.908+0000 7f3d96458640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d9006caf0 0x7f3d9006d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.908+0000 7f3d96458640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d9015bb80 con 0x7f3d9015b6a0 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.908+0000 7f3d96458640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f3d900724a0 con 0x7f3d900720c0 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.908+0000 7f3d96458640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d900b3750 con 0x7f3d9006caf0 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.908+0000 7f3d8ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d900720c0 0x7f3d900b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.908+0000 7f3d8ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d900720c0 0x7f3d900b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54638/0 (socket says 172.21.15.77:54638) 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.908+0000 7f3d8ffff640 1 -- 172.21.15.77:0/2596389571 learned_addr learned my addr 172.21.15.77:0/2596389571 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.908+0000 7f3d8f7fe640 1 --2- 172.21.15.77:0/2596389571 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d9006caf0 0x7f3d9006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.909+0000 7f3d8ffff640 1 -- 172.21.15.77:0/2596389571 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d9006caf0 msgr2=0x7f3d9006d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.909+0000 7f3d8ffff640 1 --2- 172.21.15.77:0/2596389571 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d9006caf0 0x7f3d9006d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.909+0000 7f3d8ffff640 1 -- 172.21.15.77:0/2596389571 >> v1:172.21.15.121:6789/0 conn(0x7f3d9015b6a0 legacy=0x7f3d90070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.909+0000 7f3d8ffff640 1 -- 172.21.15.77:0/2596389571 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d900b3a30 con 0x7f3d900720c0 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.909+0000 7f3d8ffff640 1 --2- 172.21.15.77:0/2596389571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d900720c0 0x7f3d900b4510 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009870 tx=0x7f3d8002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44ecced11a7dd5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.909+0000 7f3d8effd640 1 -- 172.21.15.77:0/2596389571 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d80037070 con 0x7f3d900720c0 2024-04-25T01:23:22.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.910+0000 7f3d8effd640 1 -- 172.21.15.77:0/2596389571 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d80032440 con 0x7f3d900720c0 2024-04-25T01:23:22.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.910+0000 7f3d8effd640 1 -- 172.21.15.77:0/2596389571 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d80032810 con 0x7f3d900720c0 2024-04-25T01:23:22.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.910+0000 7f3d96458640 1 -- 172.21.15.77:0/2596389571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d900720c0 msgr2=0x7f3d900b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.910+0000 7f3d96458640 1 --2- 172.21.15.77:0/2596389571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d900720c0 0x7f3d900b4510 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009870 tx=0x7f3d8002a1c0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.910+0000 7f3d96458640 1 -- 172.21.15.77:0/2596389571 shutdown_connections 2024-04-25T01:23:22.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.910+0000 7f3d96458640 1 --2- 172.21.15.77:0/2596389571 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d9006caf0 0x7f3d9006d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.910+0000 7f3d96458640 1 --2- 172.21.15.77:0/2596389571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d900720c0 0x7f3d900b4510 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.910+0000 7f3d96458640 1 -- 172.21.15.77:0/2596389571 >> 172.21.15.77:0/2596389571 conn(0x7f3d9006b720 msgr2=0x7f3d900b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:22.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.910+0000 7f3d96458640 1 -- 172.21.15.77:0/2596389571 shutdown_connections 2024-04-25T01:23:22.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.910+0000 7f3d96458640 1 -- 172.21.15.77:0/2596389571 wait complete. 2024-04-25T01:23:22.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.911+0000 7f3d96458640 1 Processor -- start 2024-04-25T01:23:22.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d96458640 1 -- start start 2024-04-25T01:23:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d96458640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d9006caf0 0x7f3d900cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d96458640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d900720c0 0x7f3d900cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d96458640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d9015b6a0 0x7f3d9014aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d96458640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3d900cf3c0 con 0x7f3d9015b6a0 2024-04-25T01:23:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d96458640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d900b3ff0 con 0x7f3d900720c0 2024-04-25T01:23:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d96458640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d900cf540 con 0x7f3d9006caf0 2024-04-25T01:23:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d949f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d9015b6a0 0x7f3d9014aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d9006caf0 0x7f3d900cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d949f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d9015b6a0 0x7f3d9014aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54644/0 (socket says 172.21.15.77:54644) 2024-04-25T01:23:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d9006caf0 0x7f3d900cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:34872/0 (socket says 172.21.15.77:34872) 2024-04-25T01:23:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d949f5640 1 -- 172.21.15.77:0/4286224108 learned_addr learned my addr 172.21.15.77:0/4286224108 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.912+0000 7f3d8f7fe640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d900720c0 0x7f3d900cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.913+0000 7f3d949f5640 1 -- 172.21.15.77:0/4286224108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d9006caf0 msgr2=0x7f3d900cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.913+0000 7f3d949f5640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d9006caf0 0x7f3d900cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.913+0000 7f3d949f5640 1 -- 172.21.15.77:0/4286224108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d900720c0 msgr2=0x7f3d900cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.913+0000 7f3d949f5640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d900720c0 0x7f3d900cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.913+0000 7f3d949f5640 1 -- 172.21.15.77:0/4286224108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d901529f0 con 0x7f3d9015b6a0 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.913+0000 7f3d949f5640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d9015b6a0 0x7f3d9014aad0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3d8800c730 tx=0x7f3d8800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.913+0000 7f3d8cff9640 1 -- 172.21.15.77:0/4286224108 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d88003b70 con 0x7f3d9015b6a0 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.913+0000 7f3d8cff9640 1 -- 172.21.15.77:0/4286224108 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d88003d10 con 0x7f3d9015b6a0 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.913+0000 7f3d96458640 1 -- 172.21.15.77:0/4286224108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9014f150 con 0x7f3d9015b6a0 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.913+0000 7f3d8cff9640 1 -- 172.21.15.77:0/4286224108 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8800aa90 con 0x7f3d9015b6a0 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.913+0000 7f3d96458640 1 -- 172.21.15.77:0/4286224108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d90152f90 con 0x7f3d9015b6a0 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.914+0000 7f3d96458640 1 -- 172.21.15.77:0/4286224108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d90068fb0 con 0x7f3d9015b6a0 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.914+0000 7f3d8cff9640 1 -- 172.21.15.77:0/4286224108 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3d8800ac30 con 0x7f3d9015b6a0 2024-04-25T01:23:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.915+0000 7f3d8cff9640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3d7c06be30 0x7f3d7c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.915+0000 7f3d8cff9640 1 -- 172.21.15.77:0/4286224108 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7f3d8808ab30 con 0x7f3d9015b6a0 2024-04-25T01:23:22.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.915+0000 7f3d8ffff640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3d7c06be30 0x7f3d7c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.915+0000 7f3d8ffff640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3d7c06be30 0x7f3d7c06e2f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3d80034000 tx=0x7f3d8002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:22.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:22.919+0000 7f3d8cff9640 1 -- 172.21.15.77:0/4286224108 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3d88058fb0 con 0x7f3d9015b6a0 2024-04-25T01:23:23.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.094+0000 7f3d96458640 1 -- 172.21.15.77:0/4286224108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f3d900b3400 con 0x7f3d9015b6a0 2024-04-25T01:23:23.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.097+0000 7f3d8cff9640 1 -- 172.21.15.77:0/4286224108 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f3d8805d2b0 con 0x7f3d9015b6a0 2024-04-25T01:23:23.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.099+0000 7f3d96458640 1 -- 172.21.15.77:0/4286224108 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3d7c06be30 msgr2=0x7f3d7c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.099+0000 7f3d96458640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3d7c06be30 0x7f3d7c06e2f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3d80034000 tx=0x7f3d8002e000 comp rx=0 tx=0).stop 2024-04-25T01:23:23.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.099+0000 7f3d96458640 1 -- 172.21.15.77:0/4286224108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d9015b6a0 msgr2=0x7f3d9014aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.099+0000 7f3d96458640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d9015b6a0 0x7f3d9014aad0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3d8800c730 tx=0x7f3d8800cbf0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.099+0000 7f3d96458640 1 -- 172.21.15.77:0/4286224108 shutdown_connections 2024-04-25T01:23:23.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.099+0000 7f3d96458640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3d7c06be30 0x7f3d7c06e2f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.099+0000 7f3d96458640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d9015b6a0 0x7f3d9014aad0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.099+0000 7f3d96458640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d900720c0 0x7f3d900cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.099+0000 7f3d96458640 1 --2- 172.21.15.77:0/4286224108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d9006caf0 0x7f3d900cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.099+0000 7f3d96458640 1 -- 172.21.15.77:0/4286224108 >> 172.21.15.77:0/4286224108 conn(0x7f3d9006b720 msgr2=0x7f3d9015cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:23.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.100+0000 7f3d96458640 1 -- 172.21.15.77:0/4286224108 shutdown_connections 2024-04-25T01:23:23.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.100+0000 7f3d96458640 1 -- 172.21.15.77:0/4286224108 wait complete. 2024-04-25T01:23:23.091 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T01:23:23.092 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:23:23.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:23:23.176 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T01:23:23.176 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:23:23.176 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:23.192 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T01:23:23.192 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:23:23.192 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:23.209 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=434 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:23:23.210 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:23:23.210 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:23.247 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T01:23:23.247 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:23:23.247 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:23.267 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=434 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:23:23.267 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T01:23:23.267 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:23.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.401+0000 7face7914640 1 Processor -- start 2024-04-25T01:23:23.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.402+0000 7face7914640 1 -- start start 2024-04-25T01:23:23.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.402+0000 7face7914640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7face00d4a10 0x7face00cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.402+0000 7face7914640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7face00b3ee0 0x7face00b4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.402+0000 7face7914640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7face00d1e90 con 0x7face015e4c0 2024-04-25T01:23:23.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.402+0000 7face7914640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7face00b54a0 con 0x7face00d4a10 2024-04-25T01:23:23.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.402+0000 7face7914640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7face00d2130 con 0x7face00b3ee0 2024-04-25T01:23:23.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.402+0000 7face4eaf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7face00d4a10 0x7face00cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.402+0000 7face4eaf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7face00d4a10 0x7face00cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54648/0 (socket says 172.21.15.77:54648) 2024-04-25T01:23:23.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.402+0000 7face4eaf640 1 -- 172.21.15.77:0/2565408812 learned_addr learned my addr 172.21.15.77:0/2565408812 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:23.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.403+0000 7face5eb1640 1 --2- 172.21.15.77:0/2565408812 >> v2:172.21.15.121:3300/0 conn(0x7face00b3ee0 0x7face00b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.403+0000 7face4eaf640 1 -- 172.21.15.77:0/2565408812 >> v2:172.21.15.121:3300/0 conn(0x7face00b3ee0 msgr2=0x7face00b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:23.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.403+0000 7face4eaf640 1 --2- 172.21.15.77:0/2565408812 >> v2:172.21.15.121:3300/0 conn(0x7face00b3ee0 0x7face00b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.403+0000 7face4eaf640 1 -- 172.21.15.77:0/2565408812 >> v1:172.21.15.121:6789/0 conn(0x7face015e4c0 legacy=0x7face00d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.403+0000 7face4eaf640 1 -- 172.21.15.77:0/2565408812 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face00b5780 con 0x7face00d4a10 2024-04-25T01:23:23.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.403+0000 7face4eaf640 1 --2- 172.21.15.77:0/2565408812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7face00d4a10 0x7face00cfaf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7face008d2e0 tx=0x7facd002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab7ed56f7a1e30c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:23.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.404+0000 7facd7fff640 1 -- 172.21.15.77:0/2565408812 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd0037070 con 0x7face00d4a10 2024-04-25T01:23:23.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.404+0000 7facd7fff640 1 -- 172.21.15.77:0/2565408812 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facd0032440 con 0x7face00d4a10 2024-04-25T01:23:23.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.404+0000 7facd7fff640 1 -- 172.21.15.77:0/2565408812 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd0032810 con 0x7face00d4a10 2024-04-25T01:23:23.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.404+0000 7face7914640 1 -- 172.21.15.77:0/2565408812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7face00d4a10 msgr2=0x7face00cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.404+0000 7face7914640 1 --2- 172.21.15.77:0/2565408812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7face00d4a10 0x7face00cfaf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7face008d2e0 tx=0x7facd002a1e0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.404+0000 7face7914640 1 -- 172.21.15.77:0/2565408812 shutdown_connections 2024-04-25T01:23:23.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.404+0000 7face7914640 1 --2- 172.21.15.77:0/2565408812 >> v2:172.21.15.121:3300/0 conn(0x7face00b3ee0 0x7face00b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.404+0000 7face7914640 1 --2- 172.21.15.77:0/2565408812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7face00d4a10 0x7face00cfaf0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.404+0000 7face7914640 1 -- 172.21.15.77:0/2565408812 >> 172.21.15.77:0/2565408812 conn(0x7face00d3ef0 msgr2=0x7face00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:23.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.404+0000 7face7914640 1 -- 172.21.15.77:0/2565408812 shutdown_connections 2024-04-25T01:23:23.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.404+0000 7face7914640 1 -- 172.21.15.77:0/2565408812 wait complete. 2024-04-25T01:23:23.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.405+0000 7face7914640 1 Processor -- start 2024-04-25T01:23:23.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.405+0000 7face7914640 1 -- start start 2024-04-25T01:23:23.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face7914640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face00b3ee0 0x7face00b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face7914640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face00d4a10 0x7face00b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face7914640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7face015e4c0 0x7face00b13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face7914640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7face00b30a0 con 0x7face015e4c0 2024-04-25T01:23:23.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face7914640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7face00b2f20 con 0x7face00d4a10 2024-04-25T01:23:23.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face7914640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7face015e8a0 con 0x7face00b3ee0 2024-04-25T01:23:23.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face5eb1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7face015e4c0 0x7face00b13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face5eb1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7face015e4c0 0x7face00b13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54656/0 (socket says 172.21.15.77:54656) 2024-04-25T01:23:23.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face5eb1640 1 -- 172.21.15.77:0/2736758803 learned_addr learned my addr 172.21.15.77:0/2736758803 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:23.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face56b0640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face00b3ee0 0x7face00b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face4eaf640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face00d4a10 0x7face00b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face5eb1640 1 -- 172.21.15.77:0/2736758803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face00b3ee0 msgr2=0x7face00b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face5eb1640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face00b3ee0 0x7face00b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face5eb1640 1 -- 172.21.15.77:0/2736758803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face00d4a10 msgr2=0x7face00b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face5eb1640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face00d4a10 0x7face00b7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face5eb1640 1 -- 172.21.15.77:0/2736758803 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face01f75e0 con 0x7face015e4c0 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.406+0000 7face56b0640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face00b3ee0 0x7face00b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.407+0000 7face5eb1640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7face015e4c0 0x7face00b13d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7facd800d5d0 tx=0x7facd800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.407+0000 7facd5ffb640 1 -- 172.21.15.77:0/2736758803 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd80125f0 con 0x7face015e4c0 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.407+0000 7facd5ffb640 1 -- 172.21.15.77:0/2736758803 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facd8019070 con 0x7face015e4c0 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.407+0000 7face7914640 1 -- 172.21.15.77:0/2736758803 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7face00b2120 con 0x7face015e4c0 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.407+0000 7facd5ffb640 1 -- 172.21.15.77:0/2736758803 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd80129f0 con 0x7face015e4c0 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.407+0000 7face7914640 1 -- 172.21.15.77:0/2736758803 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7face00b29c0 con 0x7face015e4c0 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.408+0000 7face7914640 1 -- 172.21.15.77:0/2736758803 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7face0068fb0 con 0x7face015e4c0 2024-04-25T01:23:23.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.408+0000 7facd5ffb640 1 -- 172.21.15.77:0/2736758803 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7facd8012b90 con 0x7face015e4c0 2024-04-25T01:23:23.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.408+0000 7facd5ffb640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facc406be30 0x7facc406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.409+0000 7facd5ffb640 1 -- 172.21.15.77:0/2736758803 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7facd808b070 con 0x7face015e4c0 2024-04-25T01:23:23.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.409+0000 7face56b0640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facc406be30 0x7facc406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.409+0000 7face56b0640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facc406be30 0x7facc406e2f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7facdc001e00 tx=0x7facdc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:23.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.413+0000 7facd5ffb640 1 -- 172.21.15.77:0/2736758803 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7facd8059b50 con 0x7face015e4c0 2024-04-25T01:23:23.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.623+0000 7face7914640 1 -- 172.21.15.77:0/2736758803 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7face00b4f10 con 0x7face015e4c0 2024-04-25T01:23:23.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.624+0000 7facd5ffb640 1 -- 172.21.15.77:0/2736758803 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+3875 (secure 0 0 0) 0x7facd8059450 con 0x7face015e4c0 2024-04-25T01:23:23.605 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:23.605 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1841233245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1841233245},{"type":"v1","addr":"172.21.15.121:6834","nonce":1841233245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2127648491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2127648491},{"type":"v1","addr":"172.21.15.121:6835","nonce":2127648491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2823778882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2823778882},{"type":"v1","addr":"172.21.15.77:6837","nonce":2823778882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3906326909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3906326909},{"type":"v1","addr":"172.21.15.77:6835","nonce":3906326909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[]} 2024-04-25T01:23:23.606 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 23 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 -- 172.21.15.77:0/2736758803 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facc406be30 msgr2=0x7facc406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facc406be30 0x7facc406e2f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7facdc001e00 tx=0x7facdc009040 comp rx=0 tx=0).stop 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 -- 172.21.15.77:0/2736758803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7face015e4c0 msgr2=0x7face00b13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7face015e4c0 0x7face00b13d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7facd800d5d0 tx=0x7facd800da90 comp rx=0 tx=0).stop 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 -- 172.21.15.77:0/2736758803 shutdown_connections 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facc406be30 0x7facc406e2f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7face015e4c0 0x7face00b13d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face00d4a10 0x7face00b7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 --2- 172.21.15.77:0/2736758803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face00b3ee0 0x7face00b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 -- 172.21.15.77:0/2736758803 >> 172.21.15.77:0/2736758803 conn(0x7face00d3ef0 msgr2=0x7face00d2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 -- 172.21.15.77:0/2736758803 shutdown_connections 2024-04-25T01:23:23.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.626+0000 7face7914640 1 -- 172.21.15.77:0/2736758803 wait complete. 2024-04-25T01:23:23.619 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T01:23:23.619 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T01:23:23.619 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T01:23:23.619 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:23:23.619 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T01:23:23.619 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T01:23:23.619 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T01:23:23.620 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:23:23.620 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T01:23:23.620 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T01:23:23.620 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T01:23:23.620 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:23:23.620 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T01:23:23.620 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T01:23:23.620 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T01:23:23.621 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:23:23.629 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T01:23:23.629 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T01:23:23.630 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T01:23:23.630 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T01:23:23.636 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T01:23:23.636 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T01:23:23.637 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T01:23:23.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T01:23:23.671 INFO:tasks.ceph.mds.a:Started 2024-04-25T01:23:23.671 DEBUG:teuthology.parallel:result is None 2024-04-25T01:23:23.672 INFO:tasks.ceph.mds.c:Started 2024-04-25T01:23:23.672 DEBUG:teuthology.parallel:result is None 2024-04-25T01:23:23.679 INFO:tasks.ceph.mds.d:Started 2024-04-25T01:23:23.679 DEBUG:teuthology.parallel:result is None 2024-04-25T01:23:23.680 INFO:tasks.ceph.mds.b:Started 2024-04-25T01:23:23.680 DEBUG:teuthology.parallel:result is None 2024-04-25T01:23:23.680 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T01:23:23.744 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-04-25T01:23:23.744 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-04-25T01:23:23.762 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-25T01:23:23.762 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-25T01:23:23.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.801+0000 7f21247f9640 1 Processor -- start 2024-04-25T01:23:23.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.801+0000 7f21247f9640 1 -- start start 2024-04-25T01:23:23.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.802+0000 7f21247f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f211c15e4c0 0x7f211c0d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.802+0000 7f21247f9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f211c0d1e90 con 0x7f211c0d4a10 2024-04-25T01:23:23.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.802+0000 7f21247f9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f211c0b54a0 con 0x7f211c0d2d40 2024-04-25T01:23:23.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.802+0000 7f21247f9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f211c0d2130 con 0x7f211c15e4c0 2024-04-25T01:23:23.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.802+0000 7f2121d94640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f211c0d4a10 0x7f211c0b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44322/0 (socket says 172.21.15.77:44322) 2024-04-25T01:23:23.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.802+0000 7f2121d94640 1 -- 172.21.15.77:0/2579741834 learned_addr learned my addr 172.21.15.77:0/2579741834 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:23.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.802+0000 7f2122595640 1 --2- 172.21.15.77:0/2579741834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f211c15e4c0 0x7f211c0d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.802+0000 7f2121593640 1 -- 172.21.15.77:0/2579741834 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242520337 0 0) 0x7f211c0d1e90 con 0x7f211c0d4a10 2024-04-25T01:23:23.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.802+0000 7f2121593640 1 -- 172.21.15.77:0/2579741834 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2108003520 con 0x7f211c0d4a10 2024-04-25T01:23:23.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.803+0000 7f2122595640 1 -- 172.21.15.77:0/2579741834 >> v1:172.21.15.121:6789/0 conn(0x7f211c0d2d40 legacy=0x7f211c0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.803+0000 7f2122595640 1 -- 172.21.15.77:0/2579741834 >> v1:172.21.15.77:6789/0 conn(0x7f211c0d4a10 legacy=0x7f211c0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.803+0000 7f2122595640 1 -- 172.21.15.77:0/2579741834 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f211c0b5780 con 0x7f211c15e4c0 2024-04-25T01:23:23.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.803+0000 7f2122595640 1 --2- 172.21.15.77:0/2579741834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f211c15e4c0 0x7f211c0d2800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f21180097d0 tx=0x7f211802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1be61eb7523d25d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:23.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.805+0000 7f2121593640 1 -- 172.21.15.77:0/2579741834 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2118037070 con 0x7f211c15e4c0 2024-04-25T01:23:23.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.805+0000 7f2121593640 1 -- 172.21.15.77:0/2579741834 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2118032440 con 0x7f211c15e4c0 2024-04-25T01:23:23.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.805+0000 7f2121593640 1 -- 172.21.15.77:0/2579741834 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2118032810 con 0x7f211c15e4c0 2024-04-25T01:23:23.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.806+0000 7f21247f9640 1 -- 172.21.15.77:0/2579741834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f211c15e4c0 msgr2=0x7f211c0d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.806+0000 7f21247f9640 1 --2- 172.21.15.77:0/2579741834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f211c15e4c0 0x7f211c0d2800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f21180097d0 tx=0x7f211802a180 comp rx=0 tx=0).stop 2024-04-25T01:23:23.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.806+0000 7f21247f9640 1 -- 172.21.15.77:0/2579741834 shutdown_connections 2024-04-25T01:23:23.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.806+0000 7f21247f9640 1 --2- 172.21.15.77:0/2579741834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f211c15e4c0 0x7f211c0d2800 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.806+0000 7f21247f9640 1 -- 172.21.15.77:0/2579741834 >> 172.21.15.77:0/2579741834 conn(0x7f211c0d3ef0 msgr2=0x7f211c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:23.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.806+0000 7f21247f9640 1 -- 172.21.15.77:0/2579741834 shutdown_connections 2024-04-25T01:23:23.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.806+0000 7f21247f9640 1 -- 172.21.15.77:0/2579741834 wait complete. 2024-04-25T01:23:23.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.807+0000 7f21247f9640 1 Processor -- start 2024-04-25T01:23:23.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.807+0000 7f21247f9640 1 -- start start 2024-04-25T01:23:23.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.808+0000 7f21247f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f211c0d2d40 0x7f211c14e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.808+0000 7f21247f9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f211c0d4a10 0x7f211c0b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.808+0000 7f21247f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f211c15e4c0 0x7f211c15c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.808+0000 7f21247f9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f211c15ea20 con 0x7f211c15e4c0 2024-04-25T01:23:23.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.808+0000 7f21247f9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f211c15e8a0 con 0x7f211c0d4a10 2024-04-25T01:23:23.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.808+0000 7f21247f9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f211c0b45f0 con 0x7f211c0d2d40 2024-04-25T01:23:23.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.808+0000 7f2122d96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f211c15e4c0 0x7f211c15c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.808+0000 7f2122d96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f211c15e4c0 0x7f211c15c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54710/0 (socket says 172.21.15.77:54710) 2024-04-25T01:23:23.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.808+0000 7f2122d96640 1 -- 172.21.15.77:0/2990929234 learned_addr learned my addr 172.21.15.77:0/2990929234 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:23.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.808+0000 7f2122595640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f211c0d2d40 0x7f211c14e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.808+0000 7f2121d94640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f211c0d4a10 0x7f211c0b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.809+0000 7f2122d96640 1 -- 172.21.15.77:0/2990929234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f211c0d2d40 msgr2=0x7f211c14e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.809+0000 7f2122d96640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f211c0d2d40 0x7f211c14e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.809+0000 7f2122d96640 1 -- 172.21.15.77:0/2990929234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f211c0d4a10 msgr2=0x7f211c0b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.809+0000 7f2122d96640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f211c0d4a10 0x7f211c0b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.809+0000 7f2122d96640 1 -- 172.21.15.77:0/2990929234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f211c0c8b60 con 0x7f211c15e4c0 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.809+0000 7f2122d96640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f211c15e4c0 0x7f211c15c020 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f211400c710 tx=0x7f211400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.809+0000 7f2112ffd640 1 -- 172.21.15.77:0/2990929234 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2114016020 con 0x7f211c15e4c0 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.809+0000 7f2112ffd640 1 -- 172.21.15.77:0/2990929234 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2114006d50 con 0x7f211c15e4c0 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.809+0000 7f2112ffd640 1 -- 172.21.15.77:0/2990929234 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2114003bb0 con 0x7f211c15e4c0 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.809+0000 7f21247f9640 1 -- 172.21.15.77:0/2990929234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f211c0c8e40 con 0x7f211c15e4c0 2024-04-25T01:23:23.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.809+0000 7f21247f9640 1 -- 172.21.15.77:0/2990929234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f211c0c9400 con 0x7f211c15e4c0 2024-04-25T01:23:23.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.811+0000 7f2112ffd640 1 -- 172.21.15.77:0/2990929234 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f211400a850 con 0x7f211c15e4c0 2024-04-25T01:23:23.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.811+0000 7f2110ff9640 1 -- 172.21.15.77:0/2990929234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f8005290 con 0x7f211c15e4c0 2024-04-25T01:23:23.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.811+0000 7f2112ffd640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f210006be30 0x7f210006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.811+0000 7f2112ffd640 1 -- 172.21.15.77:0/2990929234 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f211408a880 con 0x7f211c15e4c0 2024-04-25T01:23:23.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.811+0000 7f2122595640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f210006be30 0x7f210006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.811+0000 7f2122595640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f210006be30 0x7f210006e2f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2118034000 tx=0x7f211802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:23.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.815+0000 7f2112ffd640 1 -- 172.21.15.77:0/2990929234 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2114020020 con 0x7f211c15e4c0 2024-04-25T01:23:23.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-04-25T01:23:23.839 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-04-25T01:23:23.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:23.998+0000 7f2110ff9640 1 -- 172.21.15.77:0/2990929234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f20f8005850 con 0x7f211c15e4c0 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.810+0000 7f2112ffd640 1 -- 172.21.15.77:0/2990929234 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v47) ==== 97+0+0 (secure 0 0 0) 0x7f211400ce20 con 0x7f211c15e4c0 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-04-25T01:23:24.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 -- 172.21.15.77:0/2990929234 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f210006be30 msgr2=0x7f210006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f210006be30 0x7f210006e2f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2118034000 tx=0x7f211802e000 comp rx=0 tx=0).stop 2024-04-25T01:23:24.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 -- 172.21.15.77:0/2990929234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f211c15e4c0 msgr2=0x7f211c15c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f211c15e4c0 0x7f211c15c020 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f211400c710 tx=0x7f211400cbd0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 -- 172.21.15.77:0/2990929234 shutdown_connections 2024-04-25T01:23:24.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f210006be30 0x7f210006e2f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f211c15e4c0 0x7f211c15c020 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f211c0d4a10 0x7f211c0b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 --2- 172.21.15.77:0/2990929234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f211c0d2d40 0x7f211c14e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 -- 172.21.15.77:0/2990929234 >> 172.21.15.77:0/2990929234 conn(0x7f211c0d3ef0 msgr2=0x7f211c15d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:24.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 -- 172.21.15.77:0/2990929234 shutdown_connections 2024-04-25T01:23:24.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.813+0000 7f2110ff9640 1 -- 172.21.15.77:0/2990929234 wait complete. 2024-04-25T01:23:24.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T01:23:24.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.929+0000 7fc859770640 1 Processor -- start 2024-04-25T01:23:24.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.929+0000 7fc859770640 1 -- start start 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc859770640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc85406fcd0 0x7fc85415ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc859770640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc8540ade10 0x7fc85406c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc859770640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc85406c670 0x7fc854161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc859770640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fc8541589e0 con 0x7fc85406fcd0 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc859770640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc8540b3890 con 0x7fc8540ade10 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc859770640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc8540ae2c0 con 0x7fc85406c670 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc852ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc85406fcd0 0x7fc85415ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc852ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc85406fcd0 0x7fc85415ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54720/0 (socket says 172.21.15.77:54720) 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc852ffd640 1 -- 172.21.15.77:0/1243012905 learned_addr learned my addr 172.21.15.77:0/1243012905 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc8537fe640 1 --2- 172.21.15.77:0/1243012905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc85406c670 0x7fc854161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc8527fc640 1 --2- 172.21.15.77:0/1243012905 >> v2:172.21.15.121:3300/0 conn(0x7fc8540ade10 0x7fc85406c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc852ffd640 1 -- 172.21.15.77:0/1243012905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc85406c670 msgr2=0x7fc854161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:24.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc852ffd640 1 --2- 172.21.15.77:0/1243012905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc85406c670 0x7fc854161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc852ffd640 1 -- 172.21.15.77:0/1243012905 >> v2:172.21.15.121:3300/0 conn(0x7fc8540ade10 msgr2=0x7fc85406c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:24.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc852ffd640 1 --2- 172.21.15.77:0/1243012905 >> v2:172.21.15.121:3300/0 conn(0x7fc8540ade10 0x7fc85406c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.930+0000 7fc852ffd640 1 -- 172.21.15.77:0/1243012905 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8540b3560 con 0x7fc85406fcd0 2024-04-25T01:23:24.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.931+0000 7fc852ffd640 1 --2- 172.21.15.77:0/1243012905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc85406fcd0 0x7fc85415ef50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc8540724a0 tx=0x7fc84402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534d0d8b8235a8de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.931+0000 7fc851ffb640 1 -- 172.21.15.77:0/1243012905 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844036070 con 0x7fc85406fcd0 2024-04-25T01:23:24.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.931+0000 7fc851ffb640 1 -- 172.21.15.77:0/1243012905 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc844031600 con 0x7fc85406fcd0 2024-04-25T01:23:24.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.931+0000 7fc851ffb640 1 -- 172.21.15.77:0/1243012905 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8440319d0 con 0x7fc85406fcd0 2024-04-25T01:23:24.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.932+0000 7fc859770640 1 -- 172.21.15.77:0/1243012905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc85406fcd0 msgr2=0x7fc85415ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.932+0000 7fc859770640 1 --2- 172.21.15.77:0/1243012905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc85406fcd0 0x7fc85415ef50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc8540724a0 tx=0x7fc84402a610 comp rx=0 tx=0).stop 2024-04-25T01:23:24.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.932+0000 7fc859770640 1 -- 172.21.15.77:0/1243012905 shutdown_connections 2024-04-25T01:23:24.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.932+0000 7fc859770640 1 --2- 172.21.15.77:0/1243012905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc85406c670 0x7fc854161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.932+0000 7fc859770640 1 --2- 172.21.15.77:0/1243012905 >> v2:172.21.15.121:3300/0 conn(0x7fc8540ade10 0x7fc85406c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.932+0000 7fc859770640 1 --2- 172.21.15.77:0/1243012905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc85406fcd0 0x7fc85415ef50 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.932+0000 7fc859770640 1 -- 172.21.15.77:0/1243012905 >> 172.21.15.77:0/1243012905 conn(0x7fc854071550 msgr2=0x7fc854071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:24.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.932+0000 7fc859770640 1 -- 172.21.15.77:0/1243012905 shutdown_connections 2024-04-25T01:23:24.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.932+0000 7fc859770640 1 -- 172.21.15.77:0/1243012905 wait complete. 2024-04-25T01:23:24.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.933+0000 7fc859770640 1 Processor -- start 2024-04-25T01:23:24.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.933+0000 7fc859770640 1 -- start start 2024-04-25T01:23:24.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc859770640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc85406c670 0x7fc8541f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc859770640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc85406fcd0 0x7fc8540cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc859770640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8540ade10 0x7fc8541f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc859770640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc8540ae7f0 con 0x7fc85406c670 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc859770640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc85406ca90 con 0x7fc8540ade10 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc859770640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc8540aead0 con 0x7fc85406fcd0 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc852ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc85406c670 0x7fc8541f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc852ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc85406c670 0x7fc8541f3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54736/0 (socket says 172.21.15.77:54736) 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc852ffd640 1 -- 172.21.15.77:0/2236555462 learned_addr learned my addr 172.21.15.77:0/2236555462 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc8537fe640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8540ade10 0x7fc8541f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc8527fc640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc85406fcd0 0x7fc8540cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc852ffd640 1 -- 172.21.15.77:0/2236555462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc85406fcd0 msgr2=0x7fc8540cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc852ffd640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc85406fcd0 0x7fc8540cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc852ffd640 1 -- 172.21.15.77:0/2236555462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8540ade10 msgr2=0x7fc8541f49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc852ffd640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8540ade10 0x7fc8541f49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc852ffd640 1 -- 172.21.15.77:0/2236555462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc85414f680 con 0x7fc85406c670 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc8537fe640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8540ade10 0x7fc8541f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.934+0000 7fc8527fc640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc85406fcd0 0x7fc8540cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.935+0000 7fc852ffd640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc85406c670 0x7fc8541f3a80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc844034040 tx=0x7fc844004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.935+0000 7fc83b7fe640 1 -- 172.21.15.77:0/2236555462 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844036050 con 0x7fc85406c670 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.935+0000 7fc83b7fe640 1 -- 172.21.15.77:0/2236555462 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc844042070 con 0x7fc85406c670 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.935+0000 7fc859770640 1 -- 172.21.15.77:0/2236555462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc85414ee60 con 0x7fc85406c670 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.935+0000 7fc83b7fe640 1 -- 172.21.15.77:0/2236555462 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844009510 con 0x7fc85406c670 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.935+0000 7fc859770640 1 -- 172.21.15.77:0/2236555462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc85414fc00 con 0x7fc85406c670 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.936+0000 7fc859770640 1 -- 172.21.15.77:0/2236555462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc854068fb0 con 0x7fc85406c670 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.936+0000 7fc83b7fe640 1 -- 172.21.15.77:0/2236555462 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc844007c30 con 0x7fc85406c670 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.936+0000 7fc83b7fe640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc84006be30 0x7fc84006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.937+0000 7fc83b7fe640 1 -- 172.21.15.77:0/2236555462 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7fc8440abb20 con 0x7fc85406c670 2024-04-25T01:23:24.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.937+0000 7fc8527fc640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc84006be30 0x7fc84006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.937+0000 7fc8527fc640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc84006be30 0x7fc84006e2f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc848006fd0 tx=0x7fc848008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:24.941+0000 7fc83b7fe640 1 -- 172.21.15.77:0/2236555462 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc844045360 con 0x7fc85406c670 2024-04-25T01:23:25.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.178+0000 7fc859770640 1 -- 172.21.15.77:0/2236555462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fc8540675a0 con 0x7fc85406c670 2024-04-25T01:23:25.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.179+0000 7fc83b7fe640 1 -- 172.21.15.77:0/2236555462 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) ==== 80+0+5304 (secure 0 0 0) 0x7fc84407a720 con 0x7fc85406c670 2024-04-25T01:23:25.160 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:25.160 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:23:25.160 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-04-25T01:23:25.160 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.160 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmRBubJBAA1jgiWa9DTdamG6ZHPTzJHA==", 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmxEjKLxAAaPXHEf3AJlrRvnyH4OKCpg==", 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:23:25.161 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmc1hYKxAATtGeOU4Ep9JGOclelSbShw==", 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.162 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylm8tWDNhAAq3irRf+ZuOWsrOLP59WOuA==", 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDzrylmFiLEFxAAiu3CMixQObEnTzVB4+oIOg==", 2024-04-25T01:23:25.163 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQD2rylm0cMJFhAAVluT5EojBn9sI3GaJjJRqw==", 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.164 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQD5rylmklH8ExAASXBlrt1JxxxhCynyR08Isw==", 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQD8rylmKQ3jERAARBac0vky2rIDzUdr4nYn/A==", 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:23:25.165 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQACsClmaGfoGBAAtbFXxGHH/zXHHDB3UBzlTA==", 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-04-25T01:23:25.166 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQAFsClm7GCeFhAANCOnPFNJhLmUfK+Kk+zHUw==", 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQAIsClmC2hZFBAAEem007DX6DrvGHwN5YTjgw==", 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.167 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQALsClmijV3FBAANujp5DQhJDc8IYocyllBBA==", 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmsfhcOhAAYZeVeMoyKmZdON0jUeFmeQ==", 2024-04-25T01:23:25.168 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow r", 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDwrylm8sQzBxAAJNZ00VUA4tzDCB+UZb+gnQ==", 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.169 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDwrylmRdgcBBAA4/WiRMEX2/s9wbevmTPgLA==", 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDwrylmG5KlDBAA28sSTbOA+nZ1ejbMP6Mqig==", 2024-04-25T01:23:25.170 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmuucRChAAIjlRw0RadOzdjfDM5tigrQ==", 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.171 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmw1qCIRAAqlQEapgQsaTD1GhzKlJX/A==", 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvrylmqtQqHhAAaEIOEzQbQjkAPt2EIRAixQ==", 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-04-25T01:23:25.172 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-04-25T01:23:25.173 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-04-25T01:23:25.173 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-04-25T01:23:25.173 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.173 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:25.173 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-04-25T01:23:25.173 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 -- 172.21.15.77:0/2236555462 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc84006be30 msgr2=0x7fc84006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc84006be30 0x7fc84006e2f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc848006fd0 tx=0x7fc848008040 comp rx=0 tx=0).stop 2024-04-25T01:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 -- 172.21.15.77:0/2236555462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc85406c670 msgr2=0x7fc8541f3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc85406c670 0x7fc8541f3a80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc844034040 tx=0x7fc844004a40 comp rx=0 tx=0).stop 2024-04-25T01:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 -- 172.21.15.77:0/2236555462 shutdown_connections 2024-04-25T01:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc84006be30 0x7fc84006e2f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8540ade10 0x7fc8541f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc85406fcd0 0x7fc8540cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 --2- 172.21.15.77:0/2236555462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc85406c670 0x7fc8541f3a80 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 -- 172.21.15.77:0/2236555462 >> 172.21.15.77:0/2236555462 conn(0x7fc854071550 msgr2=0x7fc854151720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 -- 172.21.15.77:0/2236555462 shutdown_connections 2024-04-25T01:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.181+0000 7fc859770640 1 -- 172.21.15.77:0/2236555462 wait complete. 2024-04-25T01:23:25.176 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:25.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.291+0000 7fb04899e640 1 Processor -- start 2024-04-25T01:23:25.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.291+0000 7fb04899e640 1 -- start start 2024-04-25T01:23:25.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb04899e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb04406cc40 0x7fb04406c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb04899e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb0440ccad0 0x7fb0440ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb04899e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0440d1020 con 0x7fb0440d0a60 2024-04-25T01:23:25.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb04899e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fb0440d15e0 con 0x7fb04406cc40 2024-04-25T01:23:25.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb04899e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb0440d1300 con 0x7fb0440ccad0 2024-04-25T01:23:25.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb0437fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb04406cc40 0x7fb04406c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb0437fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb04406cc40 0x7fb04406c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54748/0 (socket says 172.21.15.77:54748) 2024-04-25T01:23:25.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb0437fe640 1 -- 172.21.15.77:0/1149444658 learned_addr learned my addr 172.21.15.77:0/1149444658 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:25.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb0437fe640 1 -- 172.21.15.77:0/1149444658 >> v2:172.21.15.121:3300/0 conn(0x7fb0440ccad0 msgr2=0x7fb0440ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb0437fe640 1 --2- 172.21.15.77:0/1149444658 >> v2:172.21.15.121:3300/0 conn(0x7fb0440ccad0 0x7fb0440ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb0437fe640 1 -- 172.21.15.77:0/1149444658 >> v1:172.21.15.121:6789/0 conn(0x7fb0440d0a60 legacy=0x7fb04406d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.292+0000 7fb0437fe640 1 -- 172.21.15.77:0/1149444658 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0440cc2d0 con 0x7fb04406cc40 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.293+0000 7fb0437fe640 1 --2- 172.21.15.77:0/1149444658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb04406cc40 0x7fb04406c440 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb034009870 tx=0x7fb03402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b137696694f6ae19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.293+0000 7fb0427fc640 1 -- 172.21.15.77:0/1149444658 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034037070 con 0x7fb04406cc40 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.293+0000 7fb0427fc640 1 -- 172.21.15.77:0/1149444658 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb03402ae40 con 0x7fb04406cc40 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.293+0000 7fb0427fc640 1 -- 172.21.15.77:0/1149444658 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034032660 con 0x7fb04406cc40 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.294+0000 7fb04899e640 1 -- 172.21.15.77:0/1149444658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb04406cc40 msgr2=0x7fb04406c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.294+0000 7fb04899e640 1 --2- 172.21.15.77:0/1149444658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb04406cc40 0x7fb04406c440 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb034009870 tx=0x7fb03402a130 comp rx=0 tx=0).stop 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.294+0000 7fb04899e640 1 -- 172.21.15.77:0/1149444658 shutdown_connections 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.294+0000 7fb04899e640 1 --2- 172.21.15.77:0/1149444658 >> v2:172.21.15.121:3300/0 conn(0x7fb0440ccad0 0x7fb0440ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.294+0000 7fb04899e640 1 --2- 172.21.15.77:0/1149444658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb04406cc40 0x7fb04406c440 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.294+0000 7fb04899e640 1 -- 172.21.15.77:0/1149444658 >> 172.21.15.77:0/1149444658 conn(0x7fb0440d1b50 msgr2=0x7fb0440d1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.294+0000 7fb04899e640 1 -- 172.21.15.77:0/1149444658 shutdown_connections 2024-04-25T01:23:25.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.294+0000 7fb04899e640 1 -- 172.21.15.77:0/1149444658 wait complete. 2024-04-25T01:23:25.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.295+0000 7fb04899e640 1 Processor -- start 2024-04-25T01:23:25.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.295+0000 7fb04899e640 1 -- start start 2024-04-25T01:23:25.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.296+0000 7fb04899e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb04406cc40 0x7fb0441570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.296+0000 7fb04899e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0440ccad0 0x7fb04409ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.296+0000 7fb04899e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440d0a60 0x7fb04409f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.296+0000 7fb04899e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb04415e450 con 0x7fb0440d0a60 2024-04-25T01:23:25.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.296+0000 7fb04899e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb04415e2d0 con 0x7fb0440ccad0 2024-04-25T01:23:25.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.296+0000 7fb04899e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb0440cd4a0 con 0x7fb04406cc40 2024-04-25T01:23:25.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.296+0000 7fb043fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440d0a60 0x7fb04409f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.296+0000 7fb043fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440d0a60 0x7fb04409f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54754/0 (socket says 172.21.15.77:54754) 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.296+0000 7fb043fff640 1 -- 172.21.15.77:0/4003365913 learned_addr learned my addr 172.21.15.77:0/4003365913 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.296+0000 7fb042ffd640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0440ccad0 0x7fb04409ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.296+0000 7fb0437fe640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb04406cc40 0x7fb0441570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.297+0000 7fb043fff640 1 -- 172.21.15.77:0/4003365913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb04406cc40 msgr2=0x7fb0441570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.297+0000 7fb043fff640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb04406cc40 0x7fb0441570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.297+0000 7fb043fff640 1 -- 172.21.15.77:0/4003365913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0440ccad0 msgr2=0x7fb04409ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.297+0000 7fb043fff640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0440ccad0 0x7fb04409ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.297+0000 7fb043fff640 1 -- 172.21.15.77:0/4003365913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0440ca3d0 con 0x7fb0440d0a60 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.297+0000 7fb043fff640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440d0a60 0x7fb04409f3a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00b9a0 tx=0x7fb03c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.297+0000 7fb02bfff640 1 -- 172.21.15.77:0/4003365913 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c019070 con 0x7fb0440d0a60 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.297+0000 7fb02bfff640 1 -- 172.21.15.77:0/4003365913 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb03c00cbc0 con 0x7fb0440d0a60 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.297+0000 7fb04899e640 1 -- 172.21.15.77:0/4003365913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0440ca6b0 con 0x7fb0440d0a60 2024-04-25T01:23:25.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.297+0000 7fb02bfff640 1 -- 172.21.15.77:0/4003365913 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c014440 con 0x7fb0440d0a60 2024-04-25T01:23:25.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.297+0000 7fb04899e640 1 -- 172.21.15.77:0/4003365913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0440cac70 con 0x7fb0440d0a60 2024-04-25T01:23:25.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.298+0000 7fb04899e640 1 -- 172.21.15.77:0/4003365913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb044075350 con 0x7fb0440d0a60 2024-04-25T01:23:25.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.298+0000 7fb02bfff640 1 -- 172.21.15.77:0/4003365913 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb03c01c070 con 0x7fb0440d0a60 2024-04-25T01:23:25.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.299+0000 7fb02bfff640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb03006be30 0x7fb03006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.299+0000 7fb02bfff640 1 -- 172.21.15.77:0/4003365913 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7fb03c08a9f0 con 0x7fb0440d0a60 2024-04-25T01:23:25.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.299+0000 7fb0437fe640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb03006be30 0x7fb03006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.299+0000 7fb0437fe640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb03006be30 0x7fb03006e2f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb034034000 tx=0x7fb03402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.303+0000 7fb02bfff640 1 -- 172.21.15.77:0/4003365913 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb03c059560 con 0x7fb0440d0a60 2024-04-25T01:23:25.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.477+0000 7fb04899e640 1 -- 172.21.15.77:0/4003365913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb04415daf0 con 0x7fb0440d0a60 2024-04-25T01:23:25.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.478+0000 7fb02bfff640 1 -- 172.21.15.77:0/4003365913 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+13164 (secure 0 0 0) 0x7fb03c058e60 con 0x7fb0440d0a60 2024-04-25T01:23:25.460 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:25.460 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":47,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:23:24.808308+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:25.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 -- 172.21.15.77:0/4003365913 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb03006be30 msgr2=0x7fb03006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb03006be30 0x7fb03006e2f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb034034000 tx=0x7fb03402e000 comp rx=0 tx=0).stop 2024-04-25T01:23:25.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 -- 172.21.15.77:0/4003365913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440d0a60 msgr2=0x7fb04409f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440d0a60 0x7fb04409f3a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00b9a0 tx=0x7fb03c00be60 comp rx=0 tx=0).stop 2024-04-25T01:23:25.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 -- 172.21.15.77:0/4003365913 shutdown_connections 2024-04-25T01:23:25.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb03006be30 0x7fb03006e2f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440d0a60 0x7fb04409f3a0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0440ccad0 0x7fb04409ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 --2- 172.21.15.77:0/4003365913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb04406cc40 0x7fb0441570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 -- 172.21.15.77:0/4003365913 >> 172.21.15.77:0/4003365913 conn(0x7fb0440d1b50 msgr2=0x7fb04415ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 -- 172.21.15.77:0/4003365913 shutdown_connections 2024-04-25T01:23:25.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.480+0000 7fb04899e640 1 -- 172.21.15.77:0/4003365913 wait complete. 2024-04-25T01:23:25.472 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:23:25.472 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:23:25.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.591+0000 7f8f60914640 1 Processor -- start 2024-04-25T01:23:25.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.591+0000 7f8f60914640 1 -- start start 2024-04-25T01:23:25.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.592+0000 7f8f60914640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f5c0d23d0 0x7f8f5c0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.592+0000 7f8f60914640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f5c0cea40 con 0x7f8f5c0b9a90 2024-04-25T01:23:25.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.592+0000 7f8f60914640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f5c0cece0 con 0x7f8f5c0d27b0 2024-04-25T01:23:25.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.592+0000 7f8f60914640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5c0cefc0 con 0x7f8f5c0d23d0 2024-04-25T01:23:25.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.592+0000 7f8f5ad9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8f5c0b9a90 0x7f8f5c0b6c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44332/0 (socket says 172.21.15.77:44332) 2024-04-25T01:23:25.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.592+0000 7f8f5ad9d640 1 -- 172.21.15.77:0/3836931750 learned_addr learned my addr 172.21.15.77:0/3836931750 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:25.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.592+0000 7f8f5a59c640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f5c0d23d0 0x7f8f5c0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1364452606 0 0) 0x7f8f5c0cea40 con 0x7f8f5c0b9a90 2024-04-25T01:23:25.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f44003520 con 0x7f8f5c0b9a90 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 687648784 0 0) 0x7f8f5c0cece0 con 0x7f8f5c0d27b0 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f5c0cea40 con 0x7f8f5c0d27b0 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1135613700 0 0) 0x7f8f44003520 con 0x7f8f5c0b9a90 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f5c0d23d0 msgr2=0x7f8f5c0d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f5c0d23d0 0x7f8f5c0d4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 >> v1:172.21.15.121:6789/0 conn(0x7f8f5c0d27b0 legacy=0x7f8f5c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5c0cf2a0 con 0x7f8f5c0b9a90 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f8f54001670 con 0x7f8f5c0b9a90 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 >> v1:172.21.15.77:6789/0 conn(0x7f8f5c0b9a90 legacy=0x7f8f5c0b6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44003b70 0x7f8f44003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f44004800 0x7f8f44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f440051c0 0x7f8f4400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c0cefc0 con 0x7f8f44004800 2024-04-25T01:23:25.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4400bf30 con 0x7f8f44003b70 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8f4400c130 con 0x7f8f440051c0 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5a59c640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f5c0d23d0 0x7f8f5c0d4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f59d9b640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f44004800 0x7f8f44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5ad9d640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f440051c0 0x7f8f4400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5a59c640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44003b70 0x7f8f44003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f59d9b640 1 -- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f440051c0 msgr2=0x7f8f4400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f59d9b640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f440051c0 0x7f8f4400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f59d9b640 1 -- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44003b70 msgr2=0x7f8f44003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f59d9b640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44003b70 0x7f8f44003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f59d9b640 1 -- 172.21.15.77:0/3836931750 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f44003520 con 0x7f8f44004800 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.593+0000 7f8f5ad9d640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f440051c0 0x7f8f4400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.594+0000 7f8f59d9b640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f44004800 0x7f8f44004c80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8f500018c0 tx=0x7f8f44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7d5ab893a8e6021 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.594+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f500014e0 con 0x7f8f44004800 2024-04-25T01:23:25.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.594+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f500052f0 con 0x7f8f44004800 2024-04-25T01:23:25.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.594+0000 7f8f5959a640 1 -- 172.21.15.77:0/3836931750 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5002ac90 con 0x7f8f44004800 2024-04-25T01:23:25.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.595+0000 7f8f60914640 1 -- 172.21.15.77:0/3836931750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f44004800 msgr2=0x7f8f44004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.595+0000 7f8f60914640 1 --2- 172.21.15.77:0/3836931750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f44004800 0x7f8f44004c80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8f500018c0 tx=0x7f8f44005830 comp rx=0 tx=0).stop 2024-04-25T01:23:25.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.595+0000 7f8f5ad9d640 1 -- 172.21.15.77:0/3836931750 reap_dead start 2024-04-25T01:23:25.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.595+0000 7f8f60914640 1 -- 172.21.15.77:0/3836931750 shutdown_connections 2024-04-25T01:23:25.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.595+0000 7f8f60914640 1 -- 172.21.15.77:0/3836931750 >> 172.21.15.77:0/3836931750 conn(0x7f8f5c0d3e30 msgr2=0x7f8f5c0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.595+0000 7f8f60914640 1 -- 172.21.15.77:0/3836931750 shutdown_connections 2024-04-25T01:23:25.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.595+0000 7f8f60914640 1 -- 172.21.15.77:0/3836931750 wait complete. 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.596+0000 7f8f60914640 1 Processor -- start 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.596+0000 7f8f60914640 1 -- start start 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f60914640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f44004800 0x7f8f5c15c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f60914640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5c1593d0 0x7f8f5c0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f60914640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f5c15e130 0x7f8f5c157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f60914640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c0cfd80 con 0x7f8f5c1593d0 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f60914640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c0cf860 con 0x7f8f5c15e130 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f60914640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5c0d4920 con 0x7f8f44004800 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f59d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5c1593d0 0x7f8f5c0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f59d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5c1593d0 0x7f8f5c0ca3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54780/0 (socket says 172.21.15.77:54780) 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f59d9b640 1 -- 172.21.15.77:0/1857622436 learned_addr learned my addr 172.21.15.77:0/1857622436 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:25.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f5ad9d640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f5c15e130 0x7f8f5c157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f5a59c640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f44004800 0x7f8f5c15c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f59d9b640 1 -- 172.21.15.77:0/1857622436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f44004800 msgr2=0x7f8f5c15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f59d9b640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f44004800 0x7f8f5c15c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f59d9b640 1 -- 172.21.15.77:0/1857622436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f5c15e130 msgr2=0x7f8f5c157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f59d9b640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f5c15e130 0x7f8f5c157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f59d9b640 1 -- 172.21.15.77:0/1857622436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5c0ca930 con 0x7f8f5c1593d0 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f59d9b640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5c1593d0 0x7f8f5c0ca3f0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8f500019e0 tx=0x7f8f500012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f4affd640 1 -- 172.21.15.77:0/1857622436 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5003f020 con 0x7f8f5c1593d0 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f4affd640 1 -- 172.21.15.77:0/1857622436 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f500054a0 con 0x7f8f5c1593d0 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.597+0000 7f8f60914640 1 -- 172.21.15.77:0/1857622436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f5c15cdc0 con 0x7f8f5c1593d0 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.598+0000 7f8f4affd640 1 -- 172.21.15.77:0/1857622436 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f50003a40 con 0x7f8f5c1593d0 2024-04-25T01:23:25.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.598+0000 7f8f60914640 1 -- 172.21.15.77:0/1857622436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f5c0c7b00 con 0x7f8f5c1593d0 2024-04-25T01:23:25.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.598+0000 7f8f60914640 1 -- 172.21.15.77:0/1857622436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f5c068fb0 con 0x7f8f5c1593d0 2024-04-25T01:23:25.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.599+0000 7f8f4affd640 1 -- 172.21.15.77:0/1857622436 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8f50042020 con 0x7f8f5c1593d0 2024-04-25T01:23:25.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.599+0000 7f8f4affd640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8f3c06be60 0x7f8f3c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.599+0000 7f8f4affd640 1 -- 172.21.15.77:0/1857622436 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7f8f500aa840 con 0x7f8f5c1593d0 2024-04-25T01:23:25.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.599+0000 7f8f5a59c640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8f3c06be60 0x7f8f3c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.600+0000 7f8f5a59c640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8f3c06be60 0x7f8f3c06e320 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c067db0 tx=0x7f8f4c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.604+0000 7f8f4affd640 1 -- 172.21.15.77:0/1857622436 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8f50049030 con 0x7f8f5c1593d0 2024-04-25T01:23:25.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.774+0000 7f8f60914640 1 -- 172.21.15.77:0/1857622436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8f5c08df50 con 0x7f8f5c1593d0 2024-04-25T01:23:25.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.775+0000 7f8f4affd640 1 -- 172.21.15.77:0/1857622436 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) ==== 93+0+10 (secure 0 0 0) 0x7f8f500793b0 con 0x7f8f5c1593d0 2024-04-25T01:23:25.756 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:23:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.776+0000 7f8f60914640 1 -- 172.21.15.77:0/1857622436 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8f3c06be60 msgr2=0x7f8f3c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.776+0000 7f8f60914640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8f3c06be60 0x7f8f3c06e320 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c067db0 tx=0x7f8f4c00b040 comp rx=0 tx=0).stop 2024-04-25T01:23:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.777+0000 7f8f60914640 1 -- 172.21.15.77:0/1857622436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5c1593d0 msgr2=0x7f8f5c0ca3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.777+0000 7f8f60914640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5c1593d0 0x7f8f5c0ca3f0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8f500019e0 tx=0x7f8f500012d0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.777+0000 7f8f60914640 1 -- 172.21.15.77:0/1857622436 shutdown_connections 2024-04-25T01:23:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.777+0000 7f8f60914640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8f3c06be60 0x7f8f3c06e320 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.777+0000 7f8f60914640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5c1593d0 0x7f8f5c0ca3f0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.777+0000 7f8f60914640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f5c15e130 0x7f8f5c157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.777+0000 7f8f60914640 1 --2- 172.21.15.77:0/1857622436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f44004800 0x7f8f5c15c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.777+0000 7f8f60914640 1 -- 172.21.15.77:0/1857622436 >> 172.21.15.77:0/1857622436 conn(0x7f8f5c0d3e30 msgr2=0x7f8f5c0caf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.777+0000 7f8f60914640 1 -- 172.21.15.77:0/1857622436 shutdown_connections 2024-04-25T01:23:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.777+0000 7f8f60914640 1 -- 172.21.15.77:0/1857622436 wait complete. 2024-04-25T01:23:25.769 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T01:23:25.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.884+0000 7f6b573f9640 1 Processor -- start 2024-04-25T01:23:25.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.884+0000 7f6b573f9640 1 -- start start 2024-04-25T01:23:25.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b573f9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6b5015e4c0 0x7f6b500d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b573f9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b500d4a10 0x7f6b500b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b573f9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b500d1e90 con 0x7f6b500d2d40 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b573f9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6b500b54a0 con 0x7f6b500d4a10 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b573f9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6b500d2130 con 0x7f6b5015e4c0 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b54994640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b500d4a10 0x7f6b500b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b55996640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6b500d2d40 0x7f6b500d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44338/0 (socket says 172.21.15.77:44338) 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b54994640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b500d4a10 0x7f6b500b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54790/0 (socket says 172.21.15.77:54790) 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b55996640 1 -- 172.21.15.77:0/2884335588 learned_addr learned my addr 172.21.15.77:0/2884335588 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b55195640 1 --2- 172.21.15.77:0/2884335588 >> v2:172.21.15.121:3300/0 conn(0x7f6b5015e4c0 0x7f6b500d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b54994640 1 -- 172.21.15.77:0/2884335588 >> v2:172.21.15.121:3300/0 conn(0x7f6b5015e4c0 msgr2=0x7f6b500d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b54994640 1 --2- 172.21.15.77:0/2884335588 >> v2:172.21.15.121:3300/0 conn(0x7f6b5015e4c0 0x7f6b500d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b54994640 1 -- 172.21.15.77:0/2884335588 >> v1:172.21.15.77:6789/0 conn(0x7f6b500d2d40 legacy=0x7f6b500d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.885+0000 7f6b54994640 1 -- 172.21.15.77:0/2884335588 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b500b5780 con 0x7f6b500d4a10 2024-04-25T01:23:25.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.886+0000 7f6b54994640 1 --2- 172.21.15.77:0/2884335588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b500d4a10 0x7f6b500b3ee0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6b50067960 tx=0x7f6b4402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f6aca7e7aaaa74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.886+0000 7f6b4ffff640 1 -- 172.21.15.77:0/2884335588 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44037070 con 0x7f6b500d4a10 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.886+0000 7f6b4ffff640 1 -- 172.21.15.77:0/2884335588 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b44032440 con 0x7f6b500d4a10 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.886+0000 7f6b4ffff640 1 -- 172.21.15.77:0/2884335588 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4402ae60 con 0x7f6b500d4a10 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.886+0000 7f6b573f9640 1 -- 172.21.15.77:0/2884335588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b500d4a10 msgr2=0x7f6b500b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.886+0000 7f6b573f9640 1 --2- 172.21.15.77:0/2884335588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b500d4a10 0x7f6b500b3ee0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6b50067960 tx=0x7f6b4402a110 comp rx=0 tx=0).stop 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.886+0000 7f6b573f9640 1 -- 172.21.15.77:0/2884335588 shutdown_connections 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.886+0000 7f6b573f9640 1 --2- 172.21.15.77:0/2884335588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b500d4a10 0x7f6b500b3ee0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.886+0000 7f6b573f9640 1 --2- 172.21.15.77:0/2884335588 >> v2:172.21.15.121:3300/0 conn(0x7f6b5015e4c0 0x7f6b500d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.886+0000 7f6b573f9640 1 -- 172.21.15.77:0/2884335588 >> 172.21.15.77:0/2884335588 conn(0x7f6b500d3ef0 msgr2=0x7f6b500d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.887+0000 7f6b573f9640 1 -- 172.21.15.77:0/2884335588 shutdown_connections 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.887+0000 7f6b573f9640 1 -- 172.21.15.77:0/2884335588 wait complete. 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.888+0000 7f6b573f9640 1 Processor -- start 2024-04-25T01:23:25.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.888+0000 7f6b573f9640 1 -- start start 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.888+0000 7f6b573f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b500d2d40 0x7f6b500cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.888+0000 7f6b573f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b500d4a10 0x7f6b500b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.888+0000 7f6b573f9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b5015e4c0 0x7f6b50146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.888+0000 7f6b573f9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5015ea20 con 0x7f6b500d4a10 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.888+0000 7f6b573f9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5015e8a0 con 0x7f6b5015e4c0 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.888+0000 7f6b573f9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6b500d1010 con 0x7f6b500d2d40 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.888+0000 7f6b54994640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b500d4a10 0x7f6b500b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.888+0000 7f6b54994640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b500d4a10 0x7f6b500b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54804/0 (socket says 172.21.15.77:54804) 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.888+0000 7f6b54994640 1 -- 172.21.15.77:0/1671021395 learned_addr learned my addr 172.21.15.77:0/1671021395 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b55195640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b500d2d40 0x7f6b500cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b55996640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b5015e4c0 0x7f6b50146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b54994640 1 -- 172.21.15.77:0/1671021395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b500d2d40 msgr2=0x7f6b500cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b54994640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b500d2d40 0x7f6b500cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b54994640 1 -- 172.21.15.77:0/1671021395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b5015e4c0 msgr2=0x7f6b50146e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b54994640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b5015e4c0 0x7f6b50146e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b54994640 1 -- 172.21.15.77:0/1671021395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b500c8fe0 con 0x7f6b500d4a10 2024-04-25T01:23:25.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b55195640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b500d2d40 0x7f6b500cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:25.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b54994640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b500d4a10 0x7f6b500b6cd0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6b44000c00 tx=0x7f6b440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b4dffb640 1 -- 172.21.15.77:0/1671021395 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44037040 con 0x7f6b500d4a10 2024-04-25T01:23:25.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b4dffb640 1 -- 172.21.15.77:0/1671021395 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b44042070 con 0x7f6b500d4a10 2024-04-25T01:23:25.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b573f9640 1 -- 172.21.15.77:0/1671021395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b500c92c0 con 0x7f6b500d4a10 2024-04-25T01:23:25.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b4dffb640 1 -- 172.21.15.77:0/1671021395 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44030bd0 con 0x7f6b500d4a10 2024-04-25T01:23:25.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.889+0000 7f6b573f9640 1 -- 172.21.15.77:0/1671021395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b500c9880 con 0x7f6b500d4a10 2024-04-25T01:23:25.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.890+0000 7f6b573f9640 1 -- 172.21.15.77:0/1671021395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b50068fb0 con 0x7f6b500d4a10 2024-04-25T01:23:25.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.890+0000 7f6b4dffb640 1 -- 172.21.15.77:0/1671021395 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6b44047460 con 0x7f6b500d4a10 2024-04-25T01:23:25.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.891+0000 7f6b4dffb640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b3c06be30 0x7f6b3c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.891+0000 7f6b4dffb640 1 -- 172.21.15.77:0/1671021395 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7f6b440ab900 con 0x7f6b500d4a10 2024-04-25T01:23:25.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.891+0000 7f6b55195640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b3c06be30 0x7f6b3c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.891+0000 7f6b55195640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b3c06be30 0x7f6b3c06e2f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6b48006fd0 tx=0x7f6b48008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:25.895+0000 7f6b4dffb640 1 -- 172.21.15.77:0/1671021395 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6b44032e20 con 0x7f6b500d4a10 2024-04-25T01:23:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.073+0000 7f6b573f9640 1 -- 172.21.15.77:0/1671021395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6b500d4470 con 0x7f6b500d4a10 2024-04-25T01:23:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.074+0000 7f6b4dffb640 1 -- 172.21.15.77:0/1671021395 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v47) ==== 84+0+65 (secure 0 0 0) 0x7f6b4407a3e0 con 0x7f6b500d4a10 2024-04-25T01:23:26.055 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:26.055 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-04-25T01:23:26.055 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-04-25T01:23:26.055 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-04-25T01:23:26.055 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-04-25T01:23:26.056 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-25T01:23:26.056 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-04-25T01:23:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 -- 172.21.15.77:0/1671021395 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b3c06be30 msgr2=0x7f6b3c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b3c06be30 0x7f6b3c06e2f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6b48006fd0 tx=0x7f6b48008040 comp rx=0 tx=0).stop 2024-04-25T01:23:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 -- 172.21.15.77:0/1671021395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b500d4a10 msgr2=0x7f6b500b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b500d4a10 0x7f6b500b6cd0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6b44000c00 tx=0x7f6b440046c0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 -- 172.21.15.77:0/1671021395 shutdown_connections 2024-04-25T01:23:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b3c06be30 0x7f6b3c06e2f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b500d4a10 0x7f6b500b6cd0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b5015e4c0 0x7f6b50146e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 --2- 172.21.15.77:0/1671021395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b500d2d40 0x7f6b500cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 -- 172.21.15.77:0/1671021395 >> 172.21.15.77:0/1671021395 conn(0x7f6b500d3ef0 msgr2=0x7f6b50074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 -- 172.21.15.77:0/1671021395 shutdown_connections 2024-04-25T01:23:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.076+0000 7f6b573f9640 1 -- 172.21.15.77:0/1671021395 wait complete. 2024-04-25T01:23:26.068 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T01:23:26.068 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T01:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.187+0000 7f8f1be53640 1 Processor -- start 2024-04-25T01:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.187+0000 7f8f1be53640 1 -- start start 2024-04-25T01:23:26.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.188+0000 7f8f1be53640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8f14154b00 0x7f8f1406dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.188+0000 7f8f1be53640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f1406e2f0 con 0x7f8f1415e600 2024-04-25T01:23:26.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.188+0000 7f8f1be53640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f1406e5d0 con 0x7f8f140ca1c0 2024-04-25T01:23:26.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.188+0000 7f8f1be53640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8f1406e8b0 con 0x7f8f14154b00 2024-04-25T01:23:26.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.188+0000 7f8f1a3f0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8f1415e600 0x7f8f140ce150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44344/0 (socket says 172.21.15.77:44344) 2024-04-25T01:23:26.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.188+0000 7f8f1a3f0640 1 -- 172.21.15.77:0/1861241013 learned_addr learned my addr 172.21.15.77:0/1861241013 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:26.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.188+0000 7f8f19bef640 1 --2- 172.21.15.77:0/1861241013 >> v2:172.21.15.121:3300/0 conn(0x7f8f14154b00 0x7f8f1406dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.188+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2979052495 0 0) 0x7f8f1406e2f0 con 0x7f8f1415e600 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f00003520 con 0x7f8f1415e600 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 562368817 0 0) 0x7f8f00003520 con 0x7f8f1415e600 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 >> v2:172.21.15.121:3300/0 conn(0x7f8f14154b00 msgr2=0x7f8f1406dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 --2- 172.21.15.77:0/1861241013 >> v2:172.21.15.121:3300/0 conn(0x7f8f14154b00 0x7f8f1406dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 >> v1:172.21.15.121:6789/0 conn(0x7f8f140ca1c0 legacy=0x7f8f1415def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1406eb90 con 0x7f8f1415e600 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f8f0c001670 con 0x7f8f1415e600 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 >> v1:172.21.15.77:6789/0 conn(0x7f8f1415e600 legacy=0x7f8f140ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 0x7f8f00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f00004800 0x7f8f00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f000051c0 0x7f8f0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1406e8b0 con 0x7f8f000039b0 2024-04-25T01:23:26.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0000c160 con 0x7f8f000051c0 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8f0000c360 con 0x7f8f00004800 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f19bef640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 0x7f8f00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f1a3f0640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f000051c0 0x7f8f0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f193ee640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f00004800 0x7f8f00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f19bef640 1 -- 172.21.15.77:0/1861241013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f00004800 msgr2=0x7f8f00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f19bef640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f00004800 0x7f8f00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f19bef640 1 -- 172.21.15.77:0/1861241013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f000051c0 msgr2=0x7f8f0000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f19bef640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f000051c0 0x7f8f0000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f19bef640 1 -- 172.21.15.77:0/1861241013 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f00003520 con 0x7f8f000039b0 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f1a3f0640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f000051c0 0x7f8f0000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.189+0000 7f8f1a3f0640 1 -- 172.21.15.77:0/1861241013 reap_dead start 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.190+0000 7f8f19bef640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 0x7f8f00003dd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009fc0 tx=0x7f8f00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568beafe9db38870 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.190+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10009c10 con 0x7f8f000039b0 2024-04-25T01:23:26.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.190+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f10036070 con 0x7f8f000039b0 2024-04-25T01:23:26.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.190+0000 7f8f18bed640 1 -- 172.21.15.77:0/1861241013 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10031880 con 0x7f8f000039b0 2024-04-25T01:23:26.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.191+0000 7f8f1be53640 1 -- 172.21.15.77:0/1861241013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 msgr2=0x7f8f00003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.191+0000 7f8f1be53640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 0x7f8f00003dd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009fc0 tx=0x7f8f00005830 comp rx=0 tx=0).stop 2024-04-25T01:23:26.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.191+0000 7f8f1be53640 1 -- 172.21.15.77:0/1861241013 shutdown_connections 2024-04-25T01:23:26.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.191+0000 7f8f1be53640 1 --2- 172.21.15.77:0/1861241013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 0x7f8f00003dd0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.191+0000 7f8f1be53640 1 -- 172.21.15.77:0/1861241013 >> 172.21.15.77:0/1861241013 conn(0x7f8f140cfae0 msgr2=0x7f8f140cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.191+0000 7f8f1be53640 1 -- 172.21.15.77:0/1861241013 shutdown_connections 2024-04-25T01:23:26.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.191+0000 7f8f1be53640 1 -- 172.21.15.77:0/1861241013 wait complete. 2024-04-25T01:23:26.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.192+0000 7f8f1be53640 1 Processor -- start 2024-04-25T01:23:26.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.192+0000 7f8f1be53640 1 -- start start 2024-04-25T01:23:26.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.192+0000 7f8f1be53640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 0x7f8f14159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f1be53640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f14159030 0x7f8f141ef640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f1be53640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f14157660 0x7f8f14158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f1be53640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1406dd70 con 0x7f8f000039b0 2024-04-25T01:23:26.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f1be53640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1406dbf0 con 0x7f8f14159030 2024-04-25T01:23:26.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f1be53640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8f1406def0 con 0x7f8f14157660 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f19bef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 0x7f8f14159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f19bef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 0x7f8f14159f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54820/0 (socket says 172.21.15.77:54820) 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f19bef640 1 -- 172.21.15.77:0/3863890853 learned_addr learned my addr 172.21.15.77:0/3863890853 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f1a3f0640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f14157660 0x7f8f14158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f193ee640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f14159030 0x7f8f141ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f19bef640 1 -- 172.21.15.77:0/3863890853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f14157660 msgr2=0x7f8f14158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f19bef640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f14157660 0x7f8f14158490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f19bef640 1 -- 172.21.15.77:0/3863890853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f14159030 msgr2=0x7f8f141ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f19bef640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f14159030 0x7f8f141ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f19bef640 1 -- 172.21.15.77:0/3863890853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f14156900 con 0x7f8f000039b0 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f1a3f0640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f14157660 0x7f8f14158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f193ee640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f14159030 0x7f8f141ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f19bef640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 0x7f8f14159f60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000e200 tx=0x7f8f10009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f0a7fc640 1 -- 172.21.15.77:0/3863890853 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10009c10 con 0x7f8f000039b0 2024-04-25T01:23:26.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f0a7fc640 1 -- 172.21.15.77:0/3863890853 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f10003990 con 0x7f8f000039b0 2024-04-25T01:23:26.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f1be53640 1 -- 172.21.15.77:0/3863890853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f14154f30 con 0x7f8f000039b0 2024-04-25T01:23:26.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f0a7fc640 1 -- 172.21.15.77:0/3863890853 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10003d80 con 0x7f8f000039b0 2024-04-25T01:23:26.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.193+0000 7f8f1be53640 1 -- 172.21.15.77:0/3863890853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f14156b80 con 0x7f8f000039b0 2024-04-25T01:23:26.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.194+0000 7f8f1be53640 1 -- 172.21.15.77:0/3863890853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f14068fb0 con 0x7f8f000039b0 2024-04-25T01:23:26.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.195+0000 7f8f0a7fc640 1 -- 172.21.15.77:0/3863890853 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8f10005d10 con 0x7f8f000039b0 2024-04-25T01:23:26.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.195+0000 7f8f0a7fc640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8ef806be30 0x7f8ef806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.195+0000 7f8f0a7fc640 1 -- 172.21.15.77:0/3863890853 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7f8f100abf00 con 0x7f8f000039b0 2024-04-25T01:23:26.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.195+0000 7f8f193ee640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8ef806be30 0x7f8ef806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.196+0000 7f8f193ee640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8ef806be30 0x7f8ef806e2f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8f1413f840 tx=0x7f8f04009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.199+0000 7f8f0a7fc640 1 -- 172.21.15.77:0/3863890853 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8f14156900 con 0x7f8f000039b0 2024-04-25T01:23:26.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.373+0000 7f8f1be53640 1 -- 172.21.15.77:0/3863890853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8f14159d40 con 0x7f8f000039b0 2024-04-25T01:23:26.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.817+0000 7f8f0a7fc640 1 -- 172.21.15.77:0/3863890853 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v48) ==== 138+0+0 (secure 0 0 0) 0x7f8f1007aa70 con 0x7f8f000039b0 2024-04-25T01:23:26.799 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-04-25T01:23:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 -- 172.21.15.77:0/3863890853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8ef806be30 msgr2=0x7f8ef806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8ef806be30 0x7f8ef806e2f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8f1413f840 tx=0x7f8f04009040 comp rx=0 tx=0).stop 2024-04-25T01:23:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 -- 172.21.15.77:0/3863890853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 msgr2=0x7f8f14159f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 0x7f8f14159f60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000e200 tx=0x7f8f10009750 comp rx=0 tx=0).stop 2024-04-25T01:23:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 -- 172.21.15.77:0/3863890853 shutdown_connections 2024-04-25T01:23:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8ef806be30 0x7f8ef806e2f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f14157660 0x7f8f14158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f14159030 0x7f8f141ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 --2- 172.21.15.77:0/3863890853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f000039b0 0x7f8f14159f60 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 -- 172.21.15.77:0/3863890853 >> 172.21.15.77:0/3863890853 conn(0x7f8f140cfae0 msgr2=0x7f8f14143520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 -- 172.21.15.77:0/3863890853 shutdown_connections 2024-04-25T01:23:26.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.820+0000 7f8f1be53640 1 -- 172.21.15.77:0/3863890853 wait complete. 2024-04-25T01:23:26.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T01:23:26.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.942+0000 7fc113f90640 1 Processor -- start 2024-04-25T01:23:26.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.942+0000 7fc113f90640 1 -- start start 2024-04-25T01:23:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc113f90640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc10c0ce470 0x7fc10c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc113f90640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc10c0cf700 con 0x7fc10c06e830 2024-04-25T01:23:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc113f90640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc10c0cd6b0 con 0x7fc10c15ae90 2024-04-25T01:23:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc113f90640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc10c0cd990 con 0x7fc10c0ce470 2024-04-25T01:23:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc11252d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc10c06e830 0x7fc10c071470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36826/0 (socket says 172.21.15.77:36826) 2024-04-25T01:23:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc11252d640 1 -- 172.21.15.77:0/935626246 learned_addr learned my addr 172.21.15.77:0/935626246 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc11152b640 1 --2- 172.21.15.77:0/935626246 >> v2:172.21.15.121:3300/0 conn(0x7fc10c0ce470 0x7fc10c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3102518559 0 0) 0x7fc10c0cf700 con 0x7fc10c06e830 2024-04-25T01:23:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0f8003520 con 0x7fc10c06e830 2024-04-25T01:23:26.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3685901729 0 0) 0x7fc0f8003520 con 0x7fc10c06e830 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 >> v2:172.21.15.121:3300/0 conn(0x7fc10c0ce470 msgr2=0x7fc10c06e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 --2- 172.21.15.77:0/935626246 >> v2:172.21.15.121:3300/0 conn(0x7fc10c0ce470 0x7fc10c06e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 >> v1:172.21.15.121:6789/0 conn(0x7fc10c15ae90 legacy=0x7fc10c0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c0cdc70 con 0x7fc10c06e830 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fc1040019f0 con 0x7fc10c06e830 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 >> v1:172.21.15.77:6789/0 conn(0x7fc10c06e830 legacy=0x7fc10c071470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 --2- 172.21.15.77:0/935626246 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f80039b0 0x7fc0f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 --2- 172.21.15.77:0/935626246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0f80043b0 0x7fc0f8004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.943+0000 7fc110d2a640 1 --2- 172.21.15.77:0/935626246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f8005080 0x7fc0f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c0cd990 con 0x7fc0f80043b0 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f800bfe0 con 0x7fc0f8005080 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc0f800c1e0 con 0x7fc0f80039b0 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc11152b640 1 --2- 172.21.15.77:0/935626246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0f80043b0 0x7fc0f8004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc111d2c640 1 --2- 172.21.15.77:0/935626246 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f80039b0 0x7fc0f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc11252d640 1 --2- 172.21.15.77:0/935626246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f8005080 0x7fc0f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc11152b640 1 -- 172.21.15.77:0/935626246 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f80039b0 msgr2=0x7fc0f8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:26.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc11152b640 1 --2- 172.21.15.77:0/935626246 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f80039b0 0x7fc0f8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc11152b640 1 -- 172.21.15.77:0/935626246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f8005080 msgr2=0x7fc0f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:26.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc11152b640 1 --2- 172.21.15.77:0/935626246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f8005080 0x7fc0f800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc11152b640 1 -- 172.21.15.77:0/935626246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f8003520 con 0x7fc0f80043b0 2024-04-25T01:23:26.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.944+0000 7fc11252d640 1 -- 172.21.15.77:0/935626246 reap_dead start 2024-04-25T01:23:26.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.945+0000 7fc11152b640 1 --2- 172.21.15.77:0/935626246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0f80043b0 0x7fc0f8004880 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009ac0 tx=0x7fc0f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd5efcca590baa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.945+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc036070 con 0x7fc0f80043b0 2024-04-25T01:23:26.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.945+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0fc031400 con 0x7fc0f80043b0 2024-04-25T01:23:26.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.945+0000 7fc110d2a640 1 -- 172.21.15.77:0/935626246 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc009690 con 0x7fc0f80043b0 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.945+0000 7fc113f90640 1 -- 172.21.15.77:0/935626246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0f80043b0 msgr2=0x7fc0f8004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.945+0000 7fc113f90640 1 --2- 172.21.15.77:0/935626246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0f80043b0 0x7fc0f8004880 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009ac0 tx=0x7fc0f8005830 comp rx=0 tx=0).stop 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.945+0000 7fc113f90640 1 -- 172.21.15.77:0/935626246 shutdown_connections 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.945+0000 7fc113f90640 1 --2- 172.21.15.77:0/935626246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0f80043b0 0x7fc0f8004880 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.945+0000 7fc113f90640 1 -- 172.21.15.77:0/935626246 >> 172.21.15.77:0/935626246 conn(0x7fc10c0cfae0 msgr2=0x7fc10c0cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.945+0000 7fc113f90640 1 -- 172.21.15.77:0/935626246 shutdown_connections 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.945+0000 7fc113f90640 1 -- 172.21.15.77:0/935626246 wait complete. 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.946+0000 7fc113f90640 1 Processor -- start 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc113f90640 1 -- start start 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc113f90640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f80043b0 0x7fc10c140510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc113f90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0b4ae0 0x7fc10c1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc113f90640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc10c155820 0x7fc10c0b2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc113f90640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c06e2f0 con 0x7fc10c0b4ae0 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc113f90640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c06cf40 con 0x7fc0f80043b0 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc113f90640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc10c06e470 con 0x7fc10c155820 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc11152b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0b4ae0 0x7fc10c1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc11152b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0b4ae0 0x7fc10c1552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42242/0 (socket says 172.21.15.77:42242) 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc11152b640 1 -- 172.21.15.77:0/1565256992 learned_addr learned my addr 172.21.15.77:0/1565256992 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc111d2c640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f80043b0 0x7fc10c140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.947+0000 7fc11252d640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc10c155820 0x7fc10c0b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.948+0000 7fc11152b640 1 -- 172.21.15.77:0/1565256992 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc10c155820 msgr2=0x7fc10c0b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.948+0000 7fc11152b640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc10c155820 0x7fc10c0b2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.948+0000 7fc11152b640 1 -- 172.21.15.77:0/1565256992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f80043b0 msgr2=0x7fc10c140510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.948+0000 7fc11152b640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f80043b0 0x7fc10c140510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.948+0000 7fc11152b640 1 -- 172.21.15.77:0/1565256992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c140a50 con 0x7fc10c0b4ae0 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.948+0000 7fc11152b640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0b4ae0 0x7fc10c1552e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009be0 tx=0x7fc0fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.948+0000 7fc1027fc640 1 -- 172.21.15.77:0/1565256992 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc03f020 con 0x7fc10c0b4ae0 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.948+0000 7fc1027fc640 1 -- 172.21.15.77:0/1565256992 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0fc037040 con 0x7fc10c0b4ae0 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.948+0000 7fc113f90640 1 -- 172.21.15.77:0/1565256992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc10c0b3570 con 0x7fc10c0b4ae0 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.948+0000 7fc1027fc640 1 -- 172.21.15.77:0/1565256992 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc036070 con 0x7fc10c0b4ae0 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.948+0000 7fc113f90640 1 -- 172.21.15.77:0/1565256992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc10c0b42e0 con 0x7fc10c0b4ae0 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.949+0000 7fc113f90640 1 -- 172.21.15.77:0/1565256992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc10c068fb0 con 0x7fc10c0b4ae0 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.949+0000 7fc1027fc640 1 -- 172.21.15.77:0/1565256992 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc0fc042020 con 0x7fc10c0b4ae0 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.950+0000 7fc1027fc640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0f006be30 0x7fc0f006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.950+0000 7fc1027fc640 1 -- 172.21.15.77:0/1565256992 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5619+0+0 (secure 0 0 0) 0x7fc0fc0aad80 con 0x7fc10c0b4ae0 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.950+0000 7fc111d2c640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0f006be30 0x7fc0f006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.950+0000 7fc111d2c640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0f006be30 0x7fc0f006e2f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc108001910 tx=0x7fc108009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:26.954+0000 7fc1027fc640 1 -- 172.21.15.77:0/1565256992 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc0fc049030 con 0x7fc10c0b4ae0 2024-04-25T01:23:27.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.127+0000 7fc113f90640 1 -- 172.21.15.77:0/1565256992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc10c155c00 con 0x7fc10c0b4ae0 2024-04-25T01:23:27.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.819+0000 7fc1027fc640 1 -- 172.21.15.77:0/1565256992 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v49) ==== 170+0+0 (secure 0 0 0) 0x7fc10c140a50 con 0x7fc10c0b4ae0 2024-04-25T01:23:27.801 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-04-25T01:23:27.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 -- 172.21.15.77:0/1565256992 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0f006be30 msgr2=0x7fc0f006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:27.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0f006be30 0x7fc0f006e2f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc108001910 tx=0x7fc108009040 comp rx=0 tx=0).stop 2024-04-25T01:23:27.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 -- 172.21.15.77:0/1565256992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0b4ae0 msgr2=0x7fc10c1552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:27.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0b4ae0 0x7fc10c1552e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009be0 tx=0x7fc0fc004730 comp rx=0 tx=0).stop 2024-04-25T01:23:27.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 -- 172.21.15.77:0/1565256992 shutdown_connections 2024-04-25T01:23:27.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc0f006be30 0x7fc0f006e2f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0b4ae0 0x7fc10c1552e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc10c155820 0x7fc10c0b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 --2- 172.21.15.77:0/1565256992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f80043b0 0x7fc10c140510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 -- 172.21.15.77:0/1565256992 >> 172.21.15.77:0/1565256992 conn(0x7fc10c0cfae0 msgr2=0x7fc10c15e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:27.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 -- 172.21.15.77:0/1565256992 shutdown_connections 2024-04-25T01:23:27.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.822+0000 7fc113f90640 1 -- 172.21.15.77:0/1565256992 wait complete. 2024-04-25T01:23:27.827 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T01:23:27.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.946+0000 7fcce43a9640 1 Processor -- start 2024-04-25T01:23:27.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.946+0000 7fcce43a9640 1 -- start start 2024-04-25T01:23:27.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.947+0000 7fcce43a9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fccdc0d23d0 0x7fccdc0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:27.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.947+0000 7fcce43a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccdc0b9a90 0x7fccdc0d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:27.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.947+0000 7fcce43a9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccdc0cea40 con 0x7fccdc0d27b0 2024-04-25T01:23:27.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.947+0000 7fcce43a9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fccdc0cece0 con 0x7fccdc0d23d0 2024-04-25T01:23:27.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.947+0000 7fcce43a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fccdc0cefc0 con 0x7fccdc0b9a90 2024-04-25T01:23:27.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.948+0000 7fcce2145640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fccdc0d23d0 0x7fccdc0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:27.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.948+0000 7fcce2145640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fccdc0d23d0 0x7fccdc0d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42256/0 (socket says 172.21.15.77:42256) 2024-04-25T01:23:27.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.948+0000 7fcce2145640 1 -- 172.21.15.77:0/166694629 learned_addr learned my addr 172.21.15.77:0/166694629 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:27.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.948+0000 7fcce1944640 1 --1- 172.21.15.77:0/166694629 >> v1:172.21.15.77:6789/0 conn(0x7fccdc0d27b0 0x7fccdc0b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36828/0 (socket says 172.21.15.77:36828) 2024-04-25T01:23:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.948+0000 7fcce2145640 1 -- 172.21.15.77:0/166694629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccdc0b9a90 msgr2=0x7fccdc0d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.948+0000 7fcce2145640 1 --2- 172.21.15.77:0/166694629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccdc0b9a90 0x7fccdc0d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.948+0000 7fcce2145640 1 -- 172.21.15.77:0/166694629 >> v1:172.21.15.77:6789/0 conn(0x7fccdc0d27b0 legacy=0x7fccdc0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.948+0000 7fcce2145640 1 -- 172.21.15.77:0/166694629 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccdc0cf2a0 con 0x7fccdc0d23d0 2024-04-25T01:23:27.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.948+0000 7fcce1143640 1 -- 172.21.15.77:0/166694629 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 229289667 0 0) 0x7fccdc0cea40 con 0x7fccdc0d27b0 2024-04-25T01:23:27.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce2145640 1 --2- 172.21.15.77:0/166694629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccdc0d23d0 0x7fccdc0d4920 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009e50 tx=0x7fccd802a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa586e6afe1367da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce1143640 1 -- 172.21.15.77:0/166694629 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd8036070 con 0x7fccdc0d23d0 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce1143640 1 -- 172.21.15.77:0/166694629 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccd8031600 con 0x7fccdc0d23d0 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce1143640 1 -- 172.21.15.77:0/166694629 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd80319d0 con 0x7fccdc0d23d0 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce43a9640 1 -- 172.21.15.77:0/166694629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccdc0d23d0 msgr2=0x7fccdc0d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce43a9640 1 --2- 172.21.15.77:0/166694629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccdc0d23d0 0x7fccdc0d4920 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009e50 tx=0x7fccd802a7a0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce43a9640 1 -- 172.21.15.77:0/166694629 shutdown_connections 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce43a9640 1 --2- 172.21.15.77:0/166694629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccdc0b9a90 0x7fccdc0d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce43a9640 1 --2- 172.21.15.77:0/166694629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccdc0d23d0 0x7fccdc0d4920 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce43a9640 1 -- 172.21.15.77:0/166694629 >> 172.21.15.77:0/166694629 conn(0x7fccdc0d3e30 msgr2=0x7fccdc0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce43a9640 1 -- 172.21.15.77:0/166694629 shutdown_connections 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.949+0000 7fcce43a9640 1 -- 172.21.15.77:0/166694629 wait complete. 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.950+0000 7fcce43a9640 1 Processor -- start 2024-04-25T01:23:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.950+0000 7fcce43a9640 1 -- start start 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.951+0000 7fcce43a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccdc0b9a90 0x7fccdc1593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.951+0000 7fcce43a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccdc0d23d0 0x7fccdc14e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.951+0000 7fcce43a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccdc0d27b0 0x7fccdc0cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.951+0000 7fcce43a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fccdc0cfd80 con 0x7fccdc0d23d0 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.951+0000 7fcce43a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fccdc0cf860 con 0x7fccdc0b9a90 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.951+0000 7fcce43a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fccdc0b6c80 con 0x7fccdc0d27b0 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.951+0000 7fcce1944640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccdc0d23d0 0x7fccdc14e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.951+0000 7fcce1944640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccdc0d23d0 0x7fccdc14e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42272/0 (socket says 172.21.15.77:42272) 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.951+0000 7fcce1944640 1 -- 172.21.15.77:0/210713689 learned_addr learned my addr 172.21.15.77:0/210713689 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.951+0000 7fcce2145640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccdc0b9a90 0x7fccdc1593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.951+0000 7fcce2946640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccdc0d27b0 0x7fccdc0cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fcce1944640 1 -- 172.21.15.77:0/210713689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccdc0d27b0 msgr2=0x7fccdc0cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fcce1944640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccdc0d27b0 0x7fccdc0cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fcce1944640 1 -- 172.21.15.77:0/210713689 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccdc0b9a90 msgr2=0x7fccdc1593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fcce1944640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccdc0b9a90 0x7fccdc1593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fcce1944640 1 -- 172.21.15.77:0/210713689 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccdc157fd0 con 0x7fccdc0d23d0 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fcce2946640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccdc0d27b0 0x7fccdc0cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fcce1944640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccdc0d23d0 0x7fccdc14e200 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c3f0 tx=0x7fcccc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fccd2ffd640 1 -- 172.21.15.77:0/210713689 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0074e0 con 0x7fccdc0d23d0 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fccd2ffd640 1 -- 172.21.15.77:0/210713689 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcccc0047e0 con 0x7fccdc0d23d0 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fccd2ffd640 1 -- 172.21.15.77:0/210713689 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc004c10 con 0x7fccdc0d23d0 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fcce43a9640 1 -- 172.21.15.77:0/210713689 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccdc0c9930 con 0x7fccdc0d23d0 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.952+0000 7fcce43a9640 1 -- 172.21.15.77:0/210713689 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccdc0c9ef0 con 0x7fccdc0d23d0 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.953+0000 7fcce43a9640 1 -- 172.21.15.77:0/210713689 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccdc068fb0 con 0x7fccdc0d23d0 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.953+0000 7fccd2ffd640 1 -- 172.21.15.77:0/210713689 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcccc011070 con 0x7fccdc0d23d0 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.954+0000 7fccd2ffd640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fccc006be30 0x7fccc006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:27.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.954+0000 7fccd2ffd640 1 -- 172.21.15.77:0/210713689 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6001+0+0 (secure 0 0 0) 0x7fcccc08ab20 con 0x7fccdc0d23d0 2024-04-25T01:23:27.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.954+0000 7fcce2145640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fccc006be30 0x7fccc006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:27.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.954+0000 7fcce2145640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fccc006be30 0x7fccc006e2f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fccd8033000 tx=0x7fccd8009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:27.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:27.958+0000 7fccd2ffd640 1 -- 172.21.15.77:0/210713689 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcccc0593a0 con 0x7fccdc0d23d0 2024-04-25T01:23:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.168+0000 7fcce43a9640 1 -- 172.21.15.77:0/210713689 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fccdc06d840 con 0x7fccdc0d23d0 2024-04-25T01:23:28.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.844+0000 7fccd2ffd640 1 -- 172.21.15.77:0/210713689 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v25) ==== 172+0+0 (secure 0 0 0) 0x7fcccc058ca0 con 0x7fccdc0d23d0 2024-04-25T01:23:28.825 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 7 and data pool 8 2024-04-25T01:23:28.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.848+0000 7fcce43a9640 1 -- 172.21.15.77:0/210713689 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fccc006be30 msgr2=0x7fccc006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:28.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.848+0000 7fcce43a9640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fccc006be30 0x7fccc006e2f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fccd8033000 tx=0x7fccd8009000 comp rx=0 tx=0).stop 2024-04-25T01:23:28.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.848+0000 7fcce43a9640 1 -- 172.21.15.77:0/210713689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccdc0d23d0 msgr2=0x7fccdc14e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:28.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.848+0000 7fcce43a9640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccdc0d23d0 0x7fccdc14e200 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c3f0 tx=0x7fcccc00c8b0 comp rx=0 tx=0).stop 2024-04-25T01:23:28.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.850+0000 7fcce43a9640 1 -- 172.21.15.77:0/210713689 shutdown_connections 2024-04-25T01:23:28.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.850+0000 7fcce43a9640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fccc006be30 0x7fccc006e2f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:28.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.850+0000 7fcce43a9640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccdc0d23d0 0x7fccdc14e200 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:28.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.850+0000 7fcce43a9640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccdc0d27b0 0x7fccdc0cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:28.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.850+0000 7fcce43a9640 1 --2- 172.21.15.77:0/210713689 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccdc0b9a90 0x7fccdc1593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:28.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.850+0000 7fcce43a9640 1 -- 172.21.15.77:0/210713689 >> 172.21.15.77:0/210713689 conn(0x7fccdc0d3e30 msgr2=0x7fccdc0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:28.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.850+0000 7fcce43a9640 1 -- 172.21.15.77:0/210713689 shutdown_connections 2024-04-25T01:23:28.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.850+0000 7fcce43a9640 1 -- 172.21.15.77:0/210713689 wait complete. 2024-04-25T01:23:28.856 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.976+0000 7f992e377640 1 Processor -- start 2024-04-25T01:23:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.976+0000 7f992e377640 1 -- start start 2024-04-25T01:23:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f992e377640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f992815f9b0 0x7f992815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f992e377640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806d270 0x7f992806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f992e377640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99280b4240 con 0x7f9928145590 2024-04-25T01:23:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f992e377640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9928160830 con 0x7f992815f9b0 2024-04-25T01:23:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f992e377640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9928160350 con 0x7f992806d270 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9927fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9928145590 0x7f99280b4a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36836/0 (socket says 172.21.15.77:36836) 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9927fff640 1 -- 172.21.15.77:0/4065785448 learned_addr learned my addr 172.21.15.77:0/4065785448 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f99277fe640 1 --2- 172.21.15.77:0/4065785448 >> v2:172.21.15.121:3300/0 conn(0x7f992815f9b0 0x7f992815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f992c914640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806d270 0x7f992806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2052764685 0 0) 0x7f99280b4240 con 0x7f9928145590 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9910003520 con 0x7f9928145590 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3844679091 0 0) 0x7f9910003520 con 0x7f9928145590 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806d270 msgr2=0x7f992806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806d270 0x7f992806d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 >> v2:172.21.15.121:3300/0 conn(0x7f992815f9b0 msgr2=0x7f992815fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 --2- 172.21.15.77:0/4065785448 >> v2:172.21.15.121:3300/0 conn(0x7f992815f9b0 0x7f992815fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99280b39a0 con 0x7f9928145590 2024-04-25T01:23:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f9918001670 con 0x7f9928145590 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f992c914640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806d270 0x7f992806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f99277fe640 1 --2- 172.21.15.77:0/4065785448 >> v2:172.21.15.121:3300/0 conn(0x7f992815f9b0 0x7f992815fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 >> v1:172.21.15.77:6789/0 conn(0x7f9928145590 legacy=0x7f99280b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99100039b0 0x7f9910003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9910004800 0x7f9910004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806d270 0x7f99100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9910004610 con 0x7f9910004800 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f991000b9f0 con 0x7f99100039b0 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.977+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f991000bbf0 con 0x7f992806d270 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.978+0000 7f99277fe640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9910004800 0x7f9910004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.978+0000 7f9927fff640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99100039b0 0x7f9910003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.978+0000 7f992c914640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806d270 0x7f99100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.978+0000 7f99277fe640 1 -- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806d270 msgr2=0x7f99100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.978+0000 7f99277fe640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806d270 0x7f99100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.978+0000 7f99277fe640 1 -- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99100039b0 msgr2=0x7f9910003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.978+0000 7f99277fe640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99100039b0 0x7f9910003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.978+0000 7f99277fe640 1 -- 172.21.15.77:0/4065785448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9910003520 con 0x7f9910004800 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.978+0000 7f99277fe640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9910004800 0x7f9910004c80 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f991c009c90 tx=0x7f9910005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a2087d1f2fbb35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.979+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c036070 con 0x7f9910004800 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.979+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f991c031400 con 0x7f9910004800 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.979+0000 7f9926ffd640 1 -- 172.21.15.77:0/4065785448 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c009860 con 0x7f9910004800 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.979+0000 7f992e377640 1 -- 172.21.15.77:0/4065785448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9910004800 msgr2=0x7f9910004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.979+0000 7f992e377640 1 --2- 172.21.15.77:0/4065785448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9910004800 0x7f9910004c80 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f991c009c90 tx=0x7f9910005830 comp rx=0 tx=0).stop 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.979+0000 7f992c914640 1 -- 172.21.15.77:0/4065785448 reap_dead start 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.979+0000 7f992e377640 1 -- 172.21.15.77:0/4065785448 shutdown_connections 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.979+0000 7f992e377640 1 -- 172.21.15.77:0/4065785448 >> 172.21.15.77:0/4065785448 conn(0x7f99280b1820 msgr2=0x7f99280b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.979+0000 7f992e377640 1 -- 172.21.15.77:0/4065785448 shutdown_connections 2024-04-25T01:23:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.979+0000 7f992e377640 1 -- 172.21.15.77:0/4065785448 wait complete. 2024-04-25T01:23:28.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.980+0000 7f992e377640 1 Processor -- start 2024-04-25T01:23:28.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.980+0000 7f992e377640 1 -- start start 2024-04-25T01:23:28.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.981+0000 7f992e377640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9910004800 0x7f99280ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:28.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.981+0000 7f992e377640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99281f3ba0 0x7f992813eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:28.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.981+0000 7f992e377640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99281f4020 0x7f992813f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:28.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.981+0000 7f992e377640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f99280b3f60 con 0x7f99281f3ba0 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.981+0000 7f992e377640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f992806cde0 con 0x7f9910004800 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.981+0000 7f992e377640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9928145970 con 0x7f99281f4020 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.981+0000 7f99277fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99281f3ba0 0x7f992813eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.981+0000 7f9927fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9910004800 0x7f99280ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.981+0000 7f99277fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99281f3ba0 0x7f992813eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42290/0 (socket says 172.21.15.77:42290) 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.981+0000 7f99277fe640 1 -- 172.21.15.77:0/2410937501 learned_addr learned my addr 172.21.15.77:0/2410937501 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.981+0000 7f992c914640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99281f4020 0x7f992813f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f99277fe640 1 -- 172.21.15.77:0/2410937501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99281f4020 msgr2=0x7f992813f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f99277fe640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99281f4020 0x7f992813f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f99277fe640 1 -- 172.21.15.77:0/2410937501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9910004800 msgr2=0x7f99280ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f99277fe640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9910004800 0x7f99280ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f99277fe640 1 -- 172.21.15.77:0/2410937501 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99280ae250 con 0x7f99281f3ba0 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f9927fff640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9910004800 0x7f99280ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f99277fe640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99281f3ba0 0x7f992813eb10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f991c000c00 tx=0x7f991c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f9924ff9640 1 -- 172.21.15.77:0/2410937501 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c040070 con 0x7f99281f3ba0 2024-04-25T01:23:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f9924ff9640 1 -- 172.21.15.77:0/2410937501 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f991c03b680 con 0x7f99281f3ba0 2024-04-25T01:23:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f992e377640 1 -- 172.21.15.77:0/2410937501 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99280ae530 con 0x7f99281f3ba0 2024-04-25T01:23:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f9924ff9640 1 -- 172.21.15.77:0/2410937501 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c036040 con 0x7f99281f3ba0 2024-04-25T01:23:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.982+0000 7f992e377640 1 -- 172.21.15.77:0/2410937501 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9928070f30 con 0x7f99281f3ba0 2024-04-25T01:23:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.983+0000 7f992e377640 1 -- 172.21.15.77:0/2410937501 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99280ae920 con 0x7f99281f3ba0 2024-04-25T01:23:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.984+0000 7f9924ff9640 1 -- 172.21.15.77:0/2410937501 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f991c043070 con 0x7f99281f3ba0 2024-04-25T01:23:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.984+0000 7f9924ff9640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f991406be30 0x7f991406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.984+0000 7f9924ff9640 1 -- 172.21.15.77:0/2410937501 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6101+0+0 (secure 0 0 0) 0x7f991c0ab720 con 0x7f99281f3ba0 2024-04-25T01:23:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.984+0000 7f9927fff640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f991406be30 0x7f991406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.984+0000 7f9927fff640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f991406be30 0x7f991406e2f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9918007fd0 tx=0x7f99180078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:28.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:28.988+0000 7f9924ff9640 1 -- 172.21.15.77:0/2410937501 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f991c079860 con 0x7f99281f3ba0 2024-04-25T01:23:29.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.164+0000 7f992e377640 1 -- 172.21.15.77:0/2410937501 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9928068fb0 con 0x7f99281f3ba0 2024-04-25T01:23:29.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.165+0000 7f9924ff9640 1 -- 172.21.15.77:0/2410937501 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16955 (secure 0 0 0) 0x7f991c07db60 con 0x7f99281f3ba0 2024-04-25T01:23:29.146 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:29.147 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":50,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:23:28.170122+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:23:26.392274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T01:23:27.146093+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 -- 172.21.15.77:0/2410937501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f991406be30 msgr2=0x7f991406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f991406be30 0x7f991406e2f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9918007fd0 tx=0x7f99180078f0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 -- 172.21.15.77:0/2410937501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99281f3ba0 msgr2=0x7f992813eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99281f3ba0 0x7f992813eb10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f991c000c00 tx=0x7f991c004730 comp rx=0 tx=0).stop 2024-04-25T01:23:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 -- 172.21.15.77:0/2410937501 shutdown_connections 2024-04-25T01:23:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f991406be30 0x7f991406e2f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99281f3ba0 0x7f992813eb10 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99281f4020 0x7f992813f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 --2- 172.21.15.77:0/2410937501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9910004800 0x7f99280ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 -- 172.21.15.77:0/2410937501 >> 172.21.15.77:0/2410937501 conn(0x7f99280b1820 msgr2=0x7f99280b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 -- 172.21.15.77:0/2410937501 shutdown_connections 2024-04-25T01:23:29.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.167+0000 7f992e377640 1 -- 172.21.15.77:0/2410937501 wait complete. 2024-04-25T01:23:29.160 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:29.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.275+0000 7f921d7d5640 1 Processor -- start 2024-04-25T01:23:29.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.276+0000 7f921d7d5640 1 -- start start 2024-04-25T01:23:29.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.276+0000 7f921d7d5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92180b0d80 0x7f92180b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.276+0000 7f921d7d5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f92180d27a0 0x7f92180cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.276+0000 7f921d7d5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92180cf9f0 con 0x7f921815c090 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.276+0000 7f921d7d5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f92180d2c30 con 0x7f92180b0d80 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.276+0000 7f921d7d5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f92180cf0c0 con 0x7f92180d27a0 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.276+0000 7f92167fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92180b0d80 0x7f92180b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.276+0000 7f92167fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92180b0d80 0x7f92180b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42292/0 (socket says 172.21.15.77:42292) 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.276+0000 7f92167fc640 1 -- 172.21.15.77:0/2978280723 learned_addr learned my addr 172.21.15.77:0/2978280723 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.277+0000 7f92177fe640 1 --2- 172.21.15.77:0/2978280723 >> v2:172.21.15.121:3300/0 conn(0x7f92180d27a0 0x7f92180cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.277+0000 7f92167fc640 1 -- 172.21.15.77:0/2978280723 >> v2:172.21.15.121:3300/0 conn(0x7f92180d27a0 msgr2=0x7f92180cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.277+0000 7f92167fc640 1 --2- 172.21.15.77:0/2978280723 >> v2:172.21.15.121:3300/0 conn(0x7f92180d27a0 0x7f92180cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.277+0000 7f92167fc640 1 -- 172.21.15.77:0/2978280723 >> v1:172.21.15.121:6789/0 conn(0x7f921815c090 legacy=0x7f92180cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.277+0000 7f92167fc640 1 -- 172.21.15.77:0/2978280723 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92180cf360 con 0x7f92180b0d80 2024-04-25T01:23:29.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.277+0000 7f92177fe640 1 --2- 172.21.15.77:0/2978280723 >> v2:172.21.15.121:3300/0 conn(0x7f92180d27a0 0x7f92180cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:29.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.277+0000 7f92167fc640 1 --2- 172.21.15.77:0/2978280723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92180b0d80 0x7f92180b1180 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f92180cde20 tx=0x7f920c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d087e6f7d27d1214 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.278+0000 7f9215ffb640 1 -- 172.21.15.77:0/2978280723 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c037070 con 0x7f92180b0d80 2024-04-25T01:23:29.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.278+0000 7f9215ffb640 1 -- 172.21.15.77:0/2978280723 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f920c02ae50 con 0x7f92180b0d80 2024-04-25T01:23:29.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.278+0000 7f9215ffb640 1 -- 172.21.15.77:0/2978280723 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c032860 con 0x7f92180b0d80 2024-04-25T01:23:29.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.278+0000 7f921d7d5640 1 -- 172.21.15.77:0/2978280723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92180b0d80 msgr2=0x7f92180b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.278+0000 7f921d7d5640 1 --2- 172.21.15.77:0/2978280723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92180b0d80 0x7f92180b1180 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f92180cde20 tx=0x7f920c02a300 comp rx=0 tx=0).stop 2024-04-25T01:23:29.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.278+0000 7f921d7d5640 1 -- 172.21.15.77:0/2978280723 shutdown_connections 2024-04-25T01:23:29.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.278+0000 7f921d7d5640 1 --2- 172.21.15.77:0/2978280723 >> v2:172.21.15.121:3300/0 conn(0x7f92180d27a0 0x7f92180cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.278+0000 7f921d7d5640 1 --2- 172.21.15.77:0/2978280723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92180b0d80 0x7f92180b1180 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.278+0000 7f921d7d5640 1 -- 172.21.15.77:0/2978280723 >> 172.21.15.77:0/2978280723 conn(0x7f92180d1c80 msgr2=0x7f92180d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:29.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.278+0000 7f921d7d5640 1 -- 172.21.15.77:0/2978280723 shutdown_connections 2024-04-25T01:23:29.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.278+0000 7f921d7d5640 1 -- 172.21.15.77:0/2978280723 wait complete. 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.279+0000 7f921d7d5640 1 Processor -- start 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.279+0000 7f921d7d5640 1 -- start start 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.280+0000 7f921d7d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92180b0d80 0x7f92181f4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.280+0000 7f921d7d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f921815c090 0x7f92180cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.280+0000 7f921d7d5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92180cb040 0x7f92181588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.280+0000 7f921d7d5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f92180d0620 con 0x7f921815c090 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.280+0000 7f921d7d5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f921815c470 con 0x7f92180cb040 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.280+0000 7f921d7d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f92180d07a0 con 0x7f92180b0d80 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.280+0000 7f92167fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f921815c090 0x7f92180cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.280+0000 7f92167fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f921815c090 0x7f92180cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42298/0 (socket says 172.21.15.77:42298) 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.280+0000 7f92167fc640 1 -- 172.21.15.77:0/1527283925 learned_addr learned my addr 172.21.15.77:0/1527283925 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.280+0000 7f9216ffd640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92180b0d80 0x7f92181f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.280+0000 7f92177fe640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92180cb040 0x7f92181588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f92167fc640 1 -- 172.21.15.77:0/1527283925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92180b0d80 msgr2=0x7f92181f4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f92167fc640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92180b0d80 0x7f92181f4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f92167fc640 1 -- 172.21.15.77:0/1527283925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92180cb040 msgr2=0x7f92181588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f92167fc640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92180cb040 0x7f92181588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f92167fc640 1 -- 172.21.15.77:0/1527283925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921806e270 con 0x7f921815c090 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f9216ffd640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92180b0d80 0x7f92181f4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f92167fc640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f921815c090 0x7f92180cab00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f920c02a340 tx=0x7f920c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f91ff7fe640 1 -- 172.21.15.77:0/1527283925 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c041070 con 0x7f921815c090 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f91ff7fe640 1 -- 172.21.15.77:0/1527283925 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f920c03c670 con 0x7f921815c090 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f91ff7fe640 1 -- 172.21.15.77:0/1527283925 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c037040 con 0x7f921815c090 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f921d7d5640 1 -- 172.21.15.77:0/1527283925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f921806d770 con 0x7f921815c090 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.281+0000 7f921d7d5640 1 -- 172.21.15.77:0/1527283925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f921806dcb0 con 0x7f921815c090 2024-04-25T01:23:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.282+0000 7f921d7d5640 1 -- 172.21.15.77:0/1527283925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f921806be10 con 0x7f921815c090 2024-04-25T01:23:29.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.282+0000 7f91ff7fe640 1 -- 172.21.15.77:0/1527283925 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f920c044070 con 0x7f921815c090 2024-04-25T01:23:29.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.283+0000 7f91ff7fe640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f920406be30 0x7f920406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.283+0000 7f91ff7fe640 1 -- 172.21.15.77:0/1527283925 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6101+0+0 (secure 0 0 0) 0x7f920c0ab660 con 0x7f921815c090 2024-04-25T01:23:29.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.283+0000 7f9216ffd640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f920406be30 0x7f920406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.283+0000 7f9216ffd640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f920406be30 0x7f920406e2f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f92180ce860 tx=0x7f9208009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.287+0000 7f91ff7fe640 1 -- 172.21.15.77:0/1527283925 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f921806be10 con 0x7f921815c090 2024-04-25T01:23:29.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.461+0000 7f921d7d5640 1 -- 172.21.15.77:0/1527283925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92180b1180 con 0x7f921815c090 2024-04-25T01:23:29.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.462+0000 7f91ff7fe640 1 -- 172.21.15.77:0/1527283925 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16955 (secure 0 0 0) 0x7f92180b1180 con 0x7f921815c090 2024-04-25T01:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":50,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:23:28.170122+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:23:26.392274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T01:23:27.146093+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:29.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.464+0000 7f921d7d5640 1 -- 172.21.15.77:0/1527283925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f920406be30 msgr2=0x7f920406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.464+0000 7f921d7d5640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f920406be30 0x7f920406e2f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f92180ce860 tx=0x7f9208009040 comp rx=0 tx=0).stop 2024-04-25T01:23:29.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.464+0000 7f921d7d5640 1 -- 172.21.15.77:0/1527283925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f921815c090 msgr2=0x7f92180cab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.464+0000 7f921d7d5640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f921815c090 0x7f92180cab00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f920c02a340 tx=0x7f920c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.464+0000 7f921d7d5640 1 -- 172.21.15.77:0/1527283925 shutdown_connections 2024-04-25T01:23:29.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.464+0000 7f921d7d5640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f920406be30 0x7f920406e2f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.464+0000 7f921d7d5640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f921815c090 0x7f92180cab00 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.464+0000 7f921d7d5640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92180cb040 0x7f92181588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.464+0000 7f921d7d5640 1 --2- 172.21.15.77:0/1527283925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92180b0d80 0x7f92181f4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.464+0000 7f921d7d5640 1 -- 172.21.15.77:0/1527283925 >> 172.21.15.77:0/1527283925 conn(0x7f92180d1c80 msgr2=0x7f92180b16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:29.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.465+0000 7f921d7d5640 1 -- 172.21.15.77:0/1527283925 shutdown_connections 2024-04-25T01:23:29.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.465+0000 7f921d7d5640 1 -- 172.21.15.77:0/1527283925 wait complete. 2024-04-25T01:23:29.457 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T01:23:29.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.576+0000 7f9c17b63640 1 Processor -- start 2024-04-25T01:23:29.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.576+0000 7f9c17b63640 1 -- start start 2024-04-25T01:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.577+0000 7f9c17b63640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c10158a70 0x7f9c100b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.577+0000 7f9c17b63640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9c100b37c0 0x7f9c100b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.577+0000 7f9c17b63640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c1015c210 con 0x7f9c1015b520 2024-04-25T01:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.577+0000 7f9c17b63640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9c100b2ce0 con 0x7f9c100b37c0 2024-04-25T01:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.577+0000 7f9c17b63640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9c100b2fc0 con 0x7f9c10158a70 2024-04-25T01:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.577+0000 7f9c158ff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c10158a70 0x7f9c100b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.577+0000 7f9c150fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9c100b37c0 0x7f9c100b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.577+0000 7f9c16100640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9c1015b520 0x7f9c1015b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:49124/0 (socket says 172.21.15.77:49124) 2024-04-25T01:23:29.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.577+0000 7f9c16100640 1 -- 172.21.15.77:0/2811720622 learned_addr learned my addr 172.21.15.77:0/2811720622 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:29.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.577+0000 7f9c158ff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c10158a70 0x7f9c100b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:37006/0 (socket says 172.21.15.77:37006) 2024-04-25T01:23:29.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.578+0000 7f9c150fe640 1 -- 172.21.15.77:0/2811720622 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c10158a70 msgr2=0x7f9c100b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:29.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.578+0000 7f9c150fe640 1 --2- 172.21.15.77:0/2811720622 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c10158a70 0x7f9c100b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.578+0000 7f9c150fe640 1 -- 172.21.15.77:0/2811720622 >> v1:172.21.15.121:6789/0 conn(0x7f9c1015b520 legacy=0x7f9c1015b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.578+0000 7f9c150fe640 1 -- 172.21.15.77:0/2811720622 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c100b32a0 con 0x7f9c100b37c0 2024-04-25T01:23:29.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.578+0000 7f9c150fe640 1 --2- 172.21.15.77:0/2811720622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c100b37c0 0x7f9c100b4600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9c000097a0 tx=0x7f9c0002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb1824a60fbeeacc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.579+0000 7f9c148fd640 1 -- 172.21.15.77:0/2811720622 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00037070 con 0x7f9c100b37c0 2024-04-25T01:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.579+0000 7f9c148fd640 1 -- 172.21.15.77:0/2811720622 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c00032440 con 0x7f9c100b37c0 2024-04-25T01:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.579+0000 7f9c148fd640 1 -- 172.21.15.77:0/2811720622 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00032810 con 0x7f9c100b37c0 2024-04-25T01:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.579+0000 7f9c17b63640 1 -- 172.21.15.77:0/2811720622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c100b37c0 msgr2=0x7f9c100b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.579+0000 7f9c17b63640 1 --2- 172.21.15.77:0/2811720622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c100b37c0 0x7f9c100b4600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9c000097a0 tx=0x7f9c0002a180 comp rx=0 tx=0).stop 2024-04-25T01:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.579+0000 7f9c17b63640 1 -- 172.21.15.77:0/2811720622 shutdown_connections 2024-04-25T01:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.579+0000 7f9c17b63640 1 --2- 172.21.15.77:0/2811720622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c100b37c0 0x7f9c100b4600 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.579+0000 7f9c17b63640 1 --2- 172.21.15.77:0/2811720622 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c10158a70 0x7f9c100b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.579+0000 7f9c17b63640 1 -- 172.21.15.77:0/2811720622 >> 172.21.15.77:0/2811720622 conn(0x7f9c1006b720 msgr2=0x7f9c100b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.579+0000 7f9c17b63640 1 -- 172.21.15.77:0/2811720622 shutdown_connections 2024-04-25T01:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.579+0000 7f9c17b63640 1 -- 172.21.15.77:0/2811720622 wait complete. 2024-04-25T01:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.580+0000 7f9c17b63640 1 Processor -- start 2024-04-25T01:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.580+0000 7f9c17b63640 1 -- start start 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c17b63640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c100b37c0 0x7f9c10143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c17b63640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c10158a70 0x7f9c100cf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c17b63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c1015b520 0x7f9c101e6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c17b63640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1006cb70 con 0x7f9c1015b520 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c17b63640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1006c9f0 con 0x7f9c10158a70 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c17b63640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9c1006ccf0 con 0x7f9c100b37c0 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c16100640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c1015b520 0x7f9c101e6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c16100640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c1015b520 0x7f9c101e6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42314/0 (socket says 172.21.15.77:42314) 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c16100640 1 -- 172.21.15.77:0/4241489155 learned_addr learned my addr 172.21.15.77:0/4241489155 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c150fe640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c10158a70 0x7f9c100cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c158ff640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c100b37c0 0x7f9c10143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c16100640 1 -- 172.21.15.77:0/4241489155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c100b37c0 msgr2=0x7f9c10143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c16100640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c100b37c0 0x7f9c10143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c16100640 1 -- 172.21.15.77:0/4241489155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c10158a70 msgr2=0x7f9c100cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c16100640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c10158a70 0x7f9c100cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.581+0000 7f9c16100640 1 -- 172.21.15.77:0/4241489155 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c100a94d0 con 0x7f9c1015b520 2024-04-25T01:23:29.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.582+0000 7f9c16100640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c1015b520 0x7f9c101e6b40 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c3c0 tx=0x7f9c0800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.582+0000 7f9c067fc640 1 -- 172.21.15.77:0/4241489155 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08017070 con 0x7f9c1015b520 2024-04-25T01:23:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.582+0000 7f9c067fc640 1 -- 172.21.15.77:0/4241489155 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c08005ce0 con 0x7f9c1015b520 2024-04-25T01:23:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.582+0000 7f9c17b63640 1 -- 172.21.15.77:0/4241489155 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c100a9790 con 0x7f9c1015b520 2024-04-25T01:23:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.582+0000 7f9c067fc640 1 -- 172.21.15.77:0/4241489155 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c080072f0 con 0x7f9c1015b520 2024-04-25T01:23:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.582+0000 7f9c17b63640 1 -- 172.21.15.77:0/4241489155 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c100a9d50 con 0x7f9c1015b520 2024-04-25T01:23:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.583+0000 7f9c17b63640 1 -- 172.21.15.77:0/4241489155 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c10068fb0 con 0x7f9c1015b520 2024-04-25T01:23:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.583+0000 7f9c067fc640 1 -- 172.21.15.77:0/4241489155 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9c0801a070 con 0x7f9c1015b520 2024-04-25T01:23:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.584+0000 7f9c067fc640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9bf406be30 0x7f9bf406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.584+0000 7f9c158ff640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9bf406be30 0x7f9bf406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.584+0000 7f9c067fc640 1 -- 172.21.15.77:0/4241489155 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6101+0+0 (secure 0 0 0) 0x7f9c0808a2b0 con 0x7f9c1015b520 2024-04-25T01:23:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.584+0000 7f9c158ff640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9bf406be30 0x7f9bf406e2f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0074a0 tx=0x7f9c0c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.588+0000 7f9c067fc640 1 -- 172.21.15.77:0/4241489155 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9c08058ad0 con 0x7f9c1015b520 2024-04-25T01:23:29.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.797+0000 7f9c17b63640 1 -- 172.21.15.77:0/4241489155 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9c100b3c70 con 0x7f9c1015b520 2024-04-25T01:23:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.856+0000 7f9c067fc640 1 -- 172.21.15.77:0/4241489155 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7f9c080583d0 con 0x7f9c1015b520 2024-04-25T01:23:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 -- 172.21.15.77:0/4241489155 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9bf406be30 msgr2=0x7f9bf406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9bf406be30 0x7f9bf406e2f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0074a0 tx=0x7f9c0c00b040 comp rx=0 tx=0).stop 2024-04-25T01:23:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 -- 172.21.15.77:0/4241489155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c1015b520 msgr2=0x7f9c101e6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c1015b520 0x7f9c101e6b40 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c3c0 tx=0x7f9c0800c880 comp rx=0 tx=0).stop 2024-04-25T01:23:29.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 -- 172.21.15.77:0/4241489155 shutdown_connections 2024-04-25T01:23:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9bf406be30 0x7f9bf406e2f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c1015b520 0x7f9c101e6b40 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c10158a70 0x7f9c100cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 --2- 172.21.15.77:0/4241489155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c100b37c0 0x7f9c10143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 -- 172.21.15.77:0/4241489155 >> 172.21.15.77:0/4241489155 conn(0x7f9c1006b720 msgr2=0x7f9c1006d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 -- 172.21.15.77:0/4241489155 shutdown_connections 2024-04-25T01:23:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.859+0000 7f9c17b63640 1 -- 172.21.15.77:0/4241489155 wait complete. 2024-04-25T01:23:29.851 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T01:23:29.851 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T01:23:29.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.968+0000 7f1a51f42640 1 Processor -- start 2024-04-25T01:23:29.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.968+0000 7f1a51f42640 1 -- start start 2024-04-25T01:23:29.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a51f42640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1a4c15fc40 0x7f1a4c1600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a51f42640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1a4c0cedc0 0x7f1a4c0cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a51f42640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a4c06d110 con 0x7f1a4c0722e0 2024-04-25T01:23:29.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a51f42640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f1a4c160aa0 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a51f42640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1a4c06da40 con 0x7f1a4c15fc40 2024-04-25T01:23:29.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a4bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1a4c0cedc0 0x7f1a4c0cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a4bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1a4c0cedc0 0x7f1a4c0cf1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42330/0 (socket says 172.21.15.77:42330) 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a4bfff640 1 -- 172.21.15.77:0/4216038613 learned_addr learned my addr 172.21.15.77:0/4216038613 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a4affd640 1 --2- 172.21.15.77:0/4216038613 >> v2:172.21.15.121:3300/0 conn(0x7f1a4c15fc40 0x7f1a4c1600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a4bfff640 1 -- 172.21.15.77:0/4216038613 >> v2:172.21.15.121:3300/0 conn(0x7f1a4c15fc40 msgr2=0x7f1a4c1600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a4bfff640 1 --2- 172.21.15.77:0/4216038613 >> v2:172.21.15.121:3300/0 conn(0x7f1a4c15fc40 0x7f1a4c1600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a4bfff640 1 -- 172.21.15.77:0/4216038613 >> v1:172.21.15.121:6789/0 conn(0x7f1a4c0722e0 legacy=0x7f1a4c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a4bfff640 1 -- 172.21.15.77:0/4216038613 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4c06d3f0 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.969+0000 7f1a4affd640 1 --2- 172.21.15.77:0/4216038613 >> v2:172.21.15.121:3300/0 conn(0x7f1a4c15fc40 0x7f1a4c1600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.970+0000 7f1a4bfff640 1 --2- 172.21.15.77:0/4216038613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a4c0cedc0 0x7f1a4c0cf1f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0ce0a0 tx=0x7f1a4402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6461e2c59b584ccd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.970+0000 7f1a4a7fc640 1 -- 172.21.15.77:0/4216038613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44036070 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.970+0000 7f1a4a7fc640 1 -- 172.21.15.77:0/4216038613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a44031600 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.970+0000 7f1a4a7fc640 1 -- 172.21.15.77:0/4216038613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a440319d0 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.971+0000 7f1a51f42640 1 -- 172.21.15.77:0/4216038613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a4c0cedc0 msgr2=0x7f1a4c0cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.971+0000 7f1a51f42640 1 --2- 172.21.15.77:0/4216038613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a4c0cedc0 0x7f1a4c0cf1f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0ce0a0 tx=0x7f1a4402a660 comp rx=0 tx=0).stop 2024-04-25T01:23:29.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.971+0000 7f1a51f42640 1 -- 172.21.15.77:0/4216038613 shutdown_connections 2024-04-25T01:23:29.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.971+0000 7f1a51f42640 1 --2- 172.21.15.77:0/4216038613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a4c0cedc0 0x7f1a4c0cf1f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.971+0000 7f1a51f42640 1 --2- 172.21.15.77:0/4216038613 >> v2:172.21.15.121:3300/0 conn(0x7f1a4c15fc40 0x7f1a4c1600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.971+0000 7f1a51f42640 1 -- 172.21.15.77:0/4216038613 >> 172.21.15.77:0/4216038613 conn(0x7f1a4c06b720 msgr2=0x7f1a4c0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.971+0000 7f1a51f42640 1 -- 172.21.15.77:0/4216038613 shutdown_connections 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.971+0000 7f1a51f42640 1 -- 172.21.15.77:0/4216038613 wait complete. 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.972+0000 7f1a51f42640 1 Processor -- start 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.972+0000 7f1a51f42640 1 -- start start 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a51f42640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a4c0722e0 0x7f1a4c148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a51f42640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a4c0cedc0 0x7f1a4c157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a51f42640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a4c15fc40 0x7f1a4c0a97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a51f42640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c06cec0 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a51f42640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c0cf730 con 0x7f1a4c0722e0 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a51f42640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1a4c070b50 con 0x7f1a4c15fc40 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a4c0cedc0 0x7f1a4c157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a4c0cedc0 0x7f1a4c157ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42340/0 (socket says 172.21.15.77:42340) 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4affd640 1 -- 172.21.15.77:0/490670323 learned_addr learned my addr 172.21.15.77:0/490670323 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4b7fe640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a4c0722e0 0x7f1a4c148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4bfff640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a4c15fc40 0x7f1a4c0a97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4affd640 1 -- 172.21.15.77:0/490670323 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a4c15fc40 msgr2=0x7f1a4c0a97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4affd640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a4c15fc40 0x7f1a4c0a97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4affd640 1 -- 172.21.15.77:0/490670323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a4c0722e0 msgr2=0x7f1a4c148450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4affd640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a4c0722e0 0x7f1a4c148450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4affd640 1 -- 172.21.15.77:0/490670323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4c15a650 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4b7fe640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a4c0722e0 0x7f1a4c148450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a4affd640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a4c0cedc0 0x7f1a4c157ee0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000c7f0 tx=0x7f1a4000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.973+0000 7f1a33fff640 1 -- 172.21.15.77:0/490670323 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40015070 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.974+0000 7f1a33fff640 1 -- 172.21.15.77:0/490670323 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a400077c0 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.974+0000 7f1a51f42640 1 -- 172.21.15.77:0/490670323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a4c15a930 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.974+0000 7f1a33fff640 1 -- 172.21.15.77:0/490670323 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40007b70 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.974+0000 7f1a51f42640 1 -- 172.21.15.77:0/490670323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a4c15aef0 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.975+0000 7f1a51f42640 1 -- 172.21.15.77:0/490670323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a4c068fb0 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.975+0000 7f1a33fff640 1 -- 172.21.15.77:0/490670323 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1a40018070 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.976+0000 7f1a33fff640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1a3806be60 0x7f1a3806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.976+0000 7f1a4b7fe640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1a3806be60 0x7f1a3806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.976+0000 7f1a33fff640 1 -- 172.21.15.77:0/490670323 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f1a4008aff0 con 0x7f1a4c0cedc0 2024-04-25T01:23:29.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.976+0000 7f1a4b7fe640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1a3806be60 0x7f1a3806e320 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c003eb0 tx=0x7f1a3c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:29.980+0000 7f1a33fff640 1 -- 172.21.15.77:0/490670323 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1a400598a0 con 0x7f1a4c0cedc0 2024-04-25T01:23:30.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.190+0000 7f1a51f42640 1 -- 172.21.15.77:0/490670323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f1a4c156210 con 0x7f1a4c0cedc0 2024-04-25T01:23:30.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-04-25T01:23:30.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-04-25T01:23:30.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.862+0000 7f1a33fff640 1 -- 172.21.15.77:0/490670323 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) ==== 122+0+0 (secure 0 0 0) 0x7f1a400591a0 con 0x7f1a4c0cedc0 2024-04-25T01:23:30.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 -- 172.21.15.77:0/490670323 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1a3806be60 msgr2=0x7f1a3806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:30.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1a3806be60 0x7f1a3806e320 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c003eb0 tx=0x7f1a3c009040 comp rx=0 tx=0).stop 2024-04-25T01:23:30.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 -- 172.21.15.77:0/490670323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a4c0cedc0 msgr2=0x7f1a4c157ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:30.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a4c0cedc0 0x7f1a4c157ee0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000c7f0 tx=0x7f1a4000ccb0 comp rx=0 tx=0).stop 2024-04-25T01:23:30.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 -- 172.21.15.77:0/490670323 shutdown_connections 2024-04-25T01:23:30.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1a3806be60 0x7f1a3806e320 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:30.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a4c0cedc0 0x7f1a4c157ee0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:30.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a4c15fc40 0x7f1a4c0a97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:30.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 --2- 172.21.15.77:0/490670323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a4c0722e0 0x7f1a4c148450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:30.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 -- 172.21.15.77:0/490670323 >> 172.21.15.77:0/490670323 conn(0x7f1a4c06b720 msgr2=0x7f1a4c0ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:30.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 -- 172.21.15.77:0/490670323 shutdown_connections 2024-04-25T01:23:30.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.864+0000 7f1a51f42640 1 -- 172.21.15.77:0/490670323 wait complete. 2024-04-25T01:23:30.857 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T01:23:30.857 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.974+0000 7fca1e33b640 1 Processor -- start 2024-04-25T01:23:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.974+0000 7fca1e33b640 1 -- start start 2024-04-25T01:23:30.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.975+0000 7fca1e33b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fca180cfaf0 0x7fca180d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:30.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.975+0000 7fca1e33b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca180d1e90 con 0x7fca180d4a10 2024-04-25T01:23:30.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.975+0000 7fca1e33b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca180b54a0 con 0x7fca1815e4c0 2024-04-25T01:23:30.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.975+0000 7fca1e33b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fca180d2130 con 0x7fca180cfaf0 2024-04-25T01:23:30.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.975+0000 7fca177fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fca180d4a10 0x7fca180b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36850/0 (socket says 172.21.15.77:36850) 2024-04-25T01:23:30.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.975+0000 7fca177fe640 1 -- 172.21.15.77:0/19719202 learned_addr learned my addr 172.21.15.77:0/19719202 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2591959353 0 0) 0x7fca180d1e90 con 0x7fca180d4a10 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca00003520 con 0x7fca180d4a10 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca1c8d8640 1 --2- 172.21.15.77:0/19719202 >> v2:172.21.15.121:3300/0 conn(0x7fca180cfaf0 0x7fca180d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3062630852 0 0) 0x7fca00003520 con 0x7fca180d4a10 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 >> v2:172.21.15.121:3300/0 conn(0x7fca180cfaf0 msgr2=0x7fca180d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 --2- 172.21.15.77:0/19719202 >> v2:172.21.15.121:3300/0 conn(0x7fca180cfaf0 0x7fca180d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 >> v1:172.21.15.121:6789/0 conn(0x7fca1815e4c0 legacy=0x7fca180d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca180b5780 con 0x7fca180d4a10 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fca0c001670 con 0x7fca180d4a10 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 >> v1:172.21.15.77:6789/0 conn(0x7fca180d4a10 legacy=0x7fca180b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca000039b0 0x7fca00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 0x7fca00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca000051c0 0x7fca0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fca180d2130 con 0x7fca00004800 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fca0000c160 con 0x7fca000051c0 2024-04-25T01:23:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fca0000c360 con 0x7fca000039b0 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca177fe640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 0x7fca00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca1c8d8640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca000051c0 0x7fca0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca17fff640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca000039b0 0x7fca00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca177fe640 1 -- 172.21.15.77:0/19719202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca000039b0 msgr2=0x7fca00003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca177fe640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca000039b0 0x7fca00003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca177fe640 1 -- 172.21.15.77:0/19719202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca000051c0 msgr2=0x7fca0000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca177fe640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca000051c0 0x7fca0000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca177fe640 1 -- 172.21.15.77:0/19719202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca00003520 con 0x7fca00004800 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca1c8d8640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca000051c0 0x7fca0000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.976+0000 7fca1c8d8640 1 -- 172.21.15.77:0/19719202 reap_dead start 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.977+0000 7fca177fe640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 0x7fca00004c80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fca180d4e50 tx=0x7fca00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6ceb38f3dd9c11 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.977+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c007d30 con 0x7fca00004800 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.977+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0c005af0 con 0x7fca00004800 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.977+0000 7fca16ffd640 1 -- 172.21.15.77:0/19719202 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c007070 con 0x7fca00004800 2024-04-25T01:23:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.978+0000 7fca1e33b640 1 -- 172.21.15.77:0/19719202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 msgr2=0x7fca00004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:30.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.978+0000 7fca1e33b640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 0x7fca00004c80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fca180d4e50 tx=0x7fca00005830 comp rx=0 tx=0).stop 2024-04-25T01:23:30.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.978+0000 7fca1e33b640 1 -- 172.21.15.77:0/19719202 shutdown_connections 2024-04-25T01:23:30.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.978+0000 7fca1e33b640 1 --2- 172.21.15.77:0/19719202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 0x7fca00004c80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:30.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.978+0000 7fca1e33b640 1 -- 172.21.15.77:0/19719202 >> 172.21.15.77:0/19719202 conn(0x7fca180d3ef0 msgr2=0x7fca180d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:30.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.978+0000 7fca1e33b640 1 -- 172.21.15.77:0/19719202 shutdown_connections 2024-04-25T01:23:30.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.978+0000 7fca1e33b640 1 -- 172.21.15.77:0/19719202 wait complete. 2024-04-25T01:23:30.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.979+0000 7fca1e33b640 1 Processor -- start 2024-04-25T01:23:30.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.979+0000 7fca1e33b640 1 -- start start 2024-04-25T01:23:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca1e33b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 0x7fca18154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca1e33b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca180cb940 0x7fca180cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca1e33b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca180c9d20 0x7fca180ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca1e33b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fca1815ea20 con 0x7fca00004800 2024-04-25T01:23:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca1e33b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fca1815e8a0 con 0x7fca180c9d20 2024-04-25T01:23:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca1e33b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fca180d2800 con 0x7fca180cb940 2024-04-25T01:23:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca17fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 0x7fca18154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca17fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 0x7fca18154f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42364/0 (socket says 172.21.15.77:42364) 2024-04-25T01:23:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca17fff640 1 -- 172.21.15.77:0/911790905 learned_addr learned my addr 172.21.15.77:0/911790905 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca177fe640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca180cb940 0x7fca180cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca1c8d8640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca180c9d20 0x7fca180ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca17fff640 1 -- 172.21.15.77:0/911790905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca180cb940 msgr2=0x7fca180cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca17fff640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca180cb940 0x7fca180cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca17fff640 1 -- 172.21.15.77:0/911790905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca180c9d20 msgr2=0x7fca180ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca17fff640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca180c9d20 0x7fca180ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca17fff640 1 -- 172.21.15.77:0/911790905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca18155490 con 0x7fca00004800 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca17fff640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 0x7fca18154f50 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fca0800e500 tx=0x7fca0800e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.980+0000 7fca14ff9640 1 -- 172.21.15.77:0/911790905 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca080053a0 con 0x7fca00004800 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.981+0000 7fca14ff9640 1 -- 172.21.15.77:0/911790905 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca08018070 con 0x7fca00004800 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.981+0000 7fca1e33b640 1 -- 172.21.15.77:0/911790905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca180cddb0 con 0x7fca00004800 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.981+0000 7fca14ff9640 1 -- 172.21.15.77:0/911790905 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08003810 con 0x7fca00004800 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.981+0000 7fca1e33b640 1 -- 172.21.15.77:0/911790905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca180cc750 con 0x7fca00004800 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.981+0000 7fca1e33b640 1 -- 172.21.15.77:0/911790905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca18068fb0 con 0x7fca00004800 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.982+0000 7fca14ff9640 1 -- 172.21.15.77:0/911790905 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fca080039b0 con 0x7fca00004800 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.982+0000 7fca14ff9640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fca0406be30 0x7fca0406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.982+0000 7fca14ff9640 1 -- 172.21.15.77:0/911790905 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fca0808ac80 con 0x7fca00004800 2024-04-25T01:23:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.982+0000 7fca177fe640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fca0406be30 0x7fca0406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.983+0000 7fca177fe640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fca0406be30 0x7fca0406e2f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fca0c001050 tx=0x7fca0c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:30.987+0000 7fca14ff9640 1 -- 172.21.15.77:0/911790905 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fca080594b0 con 0x7fca00004800 2024-04-25T01:23:31.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.198+0000 7fca1e33b640 1 -- 172.21.15.77:0/911790905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca1806bff0 con 0x7fca00004800 2024-04-25T01:23:31.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.198+0000 7fca14ff9640 1 -- 172.21.15.77:0/911790905 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5184 (secure 0 0 0) 0x7fca08058db0 con 0x7fca00004800 2024-04-25T01:23:31.179 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:31.180 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5254,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/865504322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":865504322},{"type":"v1","addr":"172.21.15.121:6835","nonce":865504322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":5886,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3189776231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3189776231},{"type":"v1","addr":"172.21.15.77:6836","nonce":3189776231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3454287131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3454287131},{"type":"v1","addr":"172.21.15.77:6837","nonce":3454287131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:23:28.170422+0000","modified":"2024-04-25T01:23:30.191645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6834/3834135103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3834135103},{"type":"v1","addr":"172.21.15.121:6834","nonce":3834135103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":3}]} 2024-04-25T01:23:31.180 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 27 2024-04-25T01:23:31.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 -- 172.21.15.77:0/911790905 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fca0406be30 msgr2=0x7fca0406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fca0406be30 0x7fca0406e2f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fca0c001050 tx=0x7fca0c009030 comp rx=0 tx=0).stop 2024-04-25T01:23:31.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 -- 172.21.15.77:0/911790905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 msgr2=0x7fca18154f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 0x7fca18154f50 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fca0800e500 tx=0x7fca0800e9c0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 -- 172.21.15.77:0/911790905 shutdown_connections 2024-04-25T01:23:31.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fca0406be30 0x7fca0406e2f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca180c9d20 0x7fca180ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca180cb940 0x7fca180cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 --2- 172.21.15.77:0/911790905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca00004800 0x7fca18154f50 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 -- 172.21.15.77:0/911790905 >> 172.21.15.77:0/911790905 conn(0x7fca180d3ef0 msgr2=0x7fca1806c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:31.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 -- 172.21.15.77:0/911790905 shutdown_connections 2024-04-25T01:23:31.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.200+0000 7fca1e33b640 1 -- 172.21.15.77:0/911790905 wait complete. 2024-04-25T01:23:31.192 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:31.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.307+0000 7f370c109640 1 Processor -- start 2024-04-25T01:23:31.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.307+0000 7f370c109640 1 -- start start 2024-04-25T01:23:31.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.308+0000 7f370c109640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3704143d20 0x7f370415ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.308+0000 7f370c109640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f370406ed60 0x7f370415f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.308+0000 7f370c109640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f370415f770 con 0x7f3704143940 2024-04-25T01:23:31.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.308+0000 7f370c109640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3704070e60 con 0x7f3704143d20 2024-04-25T01:23:31.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.308+0000 7f370c109640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3704071140 con 0x7f370406ed60 2024-04-25T01:23:31.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.308+0000 7f3709ea5640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3704143940 0x7f370406e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36864/0 (socket says 172.21.15.77:36864) 2024-04-25T01:23:31.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.308+0000 7f3709ea5640 1 -- 172.21.15.77:0/4096606752 learned_addr learned my addr 172.21.15.77:0/4096606752 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:31.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.308+0000 7f37096a4640 1 --2- 172.21.15.77:0/4096606752 >> v2:172.21.15.121:3300/0 conn(0x7f3704143d20 0x7f370415ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.308+0000 7f370a6a6640 1 --2- 172.21.15.77:0/4096606752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f370406ed60 0x7f370415f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.308+0000 7f3708ea3640 1 -- 172.21.15.77:0/4096606752 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525039812 0 0) 0x7f370415f770 con 0x7f3704143940 2024-04-25T01:23:31.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.308+0000 7f3708ea3640 1 -- 172.21.15.77:0/4096606752 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36f0003520 con 0x7f3704143940 2024-04-25T01:23:31.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.309+0000 7f3708ea3640 1 -- 172.21.15.77:0/4096606752 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1293191112 0 0) 0x7f36f0003520 con 0x7f3704143940 2024-04-25T01:23:31.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.309+0000 7f37096a4640 1 -- 172.21.15.77:0/4096606752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f370406ed60 msgr2=0x7f370415f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:31.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.309+0000 7f37096a4640 1 --2- 172.21.15.77:0/4096606752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f370406ed60 0x7f370415f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.309+0000 7f37096a4640 1 -- 172.21.15.77:0/4096606752 >> v1:172.21.15.77:6789/0 conn(0x7f3704143940 legacy=0x7f370406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.309+0000 7f37096a4640 1 -- 172.21.15.77:0/4096606752 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3704071420 con 0x7f3704143d20 2024-04-25T01:23:31.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.309+0000 7f37096a4640 1 --2- 172.21.15.77:0/4096606752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3704143d20 0x7f370415ecf0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f36f40097d0 tx=0x7f36f402a140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ef09edc52da8bb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:31.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.309+0000 7f3708ea3640 1 -- 172.21.15.77:0/4096606752 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4037070 con 0x7f3704143d20 2024-04-25T01:23:31.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.309+0000 7f3708ea3640 1 -- 172.21.15.77:0/4096606752 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f402abd0 con 0x7f3704143d20 2024-04-25T01:23:31.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.309+0000 7f3708ea3640 1 -- 172.21.15.77:0/4096606752 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37000019f0 con 0x7f3704143d20 2024-04-25T01:23:31.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.310+0000 7f370c109640 1 -- 172.21.15.77:0/4096606752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3704143d20 msgr2=0x7f370415ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.310+0000 7f370c109640 1 --2- 172.21.15.77:0/4096606752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3704143d20 0x7f370415ecf0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f36f40097d0 tx=0x7f36f402a140 comp rx=0 tx=0).stop 2024-04-25T01:23:31.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.310+0000 7f370c109640 1 -- 172.21.15.77:0/4096606752 shutdown_connections 2024-04-25T01:23:31.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.310+0000 7f370c109640 1 --2- 172.21.15.77:0/4096606752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f370406ed60 0x7f370415f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.310+0000 7f370c109640 1 --2- 172.21.15.77:0/4096606752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3704143d20 0x7f370415ecf0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.310+0000 7f370c109640 1 -- 172.21.15.77:0/4096606752 >> 172.21.15.77:0/4096606752 conn(0x7f37040b1820 msgr2=0x7f37040b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:31.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.310+0000 7f370c109640 1 -- 172.21.15.77:0/4096606752 shutdown_connections 2024-04-25T01:23:31.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.310+0000 7f370c109640 1 -- 172.21.15.77:0/4096606752 wait complete. 2024-04-25T01:23:31.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.311+0000 7f370c109640 1 Processor -- start 2024-04-25T01:23:31.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.311+0000 7f370c109640 1 -- start start 2024-04-25T01:23:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f370c109640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370406ed60 0x7f37041563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f370c109640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3704143940 0x7f37040ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f370c109640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3704143d20 0x7f37040ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f370c109640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3704071da0 con 0x7f370406ed60 2024-04-25T01:23:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f370c109640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3704071c20 con 0x7f3704143d20 2024-04-25T01:23:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f370c109640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3704071f20 con 0x7f3704143940 2024-04-25T01:23:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f3709ea5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370406ed60 0x7f37041563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f3709ea5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370406ed60 0x7f37041563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42372/0 (socket says 172.21.15.77:42372) 2024-04-25T01:23:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f3709ea5640 1 -- 172.21.15.77:0/2663090047 learned_addr learned my addr 172.21.15.77:0/2663090047 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f37096a4640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3704143940 0x7f37040ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f370a6a6640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3704143d20 0x7f37040ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f3709ea5640 1 -- 172.21.15.77:0/2663090047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3704143940 msgr2=0x7f37040ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f3709ea5640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3704143940 0x7f37040ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f3709ea5640 1 -- 172.21.15.77:0/2663090047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3704143d20 msgr2=0x7f37040ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f3709ea5640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3704143d20 0x7f37040ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.312+0000 7f3709ea5640 1 -- 172.21.15.77:0/2663090047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3704154870 con 0x7f370406ed60 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.313+0000 7f3709ea5640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370406ed60 0x7f37041563c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3700001900 tx=0x7f370000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.313+0000 7f36fa7fc640 1 -- 172.21.15.77:0/2663090047 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700017070 con 0x7f370406ed60 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.313+0000 7f370c109640 1 -- 172.21.15.77:0/2663090047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3704154b50 con 0x7f370406ed60 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.313+0000 7f36fa7fc640 1 -- 172.21.15.77:0/2663090047 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3700012400 con 0x7f370406ed60 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.313+0000 7f36fa7fc640 1 -- 172.21.15.77:0/2663090047 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37000127f0 con 0x7f370406ed60 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.313+0000 7f370c109640 1 -- 172.21.15.77:0/2663090047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3704155110 con 0x7f370406ed60 2024-04-25T01:23:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.314+0000 7f370c109640 1 -- 172.21.15.77:0/2663090047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3704068fb0 con 0x7f370406ed60 2024-04-25T01:23:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.314+0000 7f36fa7fc640 1 -- 172.21.15.77:0/2663090047 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f370001a070 con 0x7f370406ed60 2024-04-25T01:23:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.315+0000 7f36fa7fc640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f36e806be30 0x7f36e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.315+0000 7f36fa7fc640 1 -- 172.21.15.77:0/2663090047 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f370008b0b0 con 0x7f370406ed60 2024-04-25T01:23:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.315+0000 7f37096a4640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f36e806be30 0x7f36e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.315+0000 7f37096a4640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f36e806be30 0x7f36e806e2f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f36f4034000 tx=0x7f36f402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:31.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.319+0000 7f36fa7fc640 1 -- 172.21.15.77:0/2663090047 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3700059850 con 0x7f370406ed60 2024-04-25T01:23:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.494+0000 7f370c109640 1 -- 172.21.15.77:0/2663090047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f370406d150 con 0x7f370406ed60 2024-04-25T01:23:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.495+0000 7f36fa7fc640 1 -- 172.21.15.77:0/2663090047 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+16942 (secure 0 0 0) 0x7f3700059150 con 0x7f370406ed60 2024-04-25T01:23:31.477 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:31.477 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":51,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:23:29.853531+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:23:26.392274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T01:23:27.146093+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 -- 172.21.15.77:0/2663090047 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f36e806be30 msgr2=0x7f36e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f36e806be30 0x7f36e806e2f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f36f4034000 tx=0x7f36f402e000 comp rx=0 tx=0).stop 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 -- 172.21.15.77:0/2663090047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370406ed60 msgr2=0x7f37041563c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370406ed60 0x7f37041563c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3700001900 tx=0x7f370000a430 comp rx=0 tx=0).stop 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 -- 172.21.15.77:0/2663090047 shutdown_connections 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f36e806be30 0x7f36e806e2f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3704143d20 0x7f37040ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3704143940 0x7f37040ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 --2- 172.21.15.77:0/2663090047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370406ed60 0x7f37041563c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 -- 172.21.15.77:0/2663090047 >> 172.21.15.77:0/2663090047 conn(0x7f37040b1820 msgr2=0x7f3704153210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 -- 172.21.15.77:0/2663090047 shutdown_connections 2024-04-25T01:23:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.497+0000 7f370c109640 1 -- 172.21.15.77:0/2663090047 wait complete. 2024-04-25T01:23:31.489 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T01:23:31.490 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:23:31.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.604+0000 7f045ba37640 1 Processor -- start 2024-04-25T01:23:31.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.605+0000 7f045ba37640 1 -- start start 2024-04-25T01:23:31.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.605+0000 7f045ba37640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0454143940 0x7f045406e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.605+0000 7f045ba37640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0454143d20 0x7f045406eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.605+0000 7f045ba37640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0454070e60 con 0x7f045415ecf0 2024-04-25T01:23:31.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.605+0000 7f045ba37640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f0454071140 con 0x7f0454143d20 2024-04-25T01:23:31.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.605+0000 7f045ba37640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0454071420 con 0x7f0454143940 2024-04-25T01:23:31.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.605+0000 7f0458fd2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0454143d20 0x7f045406eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.606+0000 7f0458fd2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0454143d20 0x7f045406eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42376/0 (socket says 172.21.15.77:42376) 2024-04-25T01:23:31.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.606+0000 7f0458fd2640 1 -- 172.21.15.77:0/1653412971 learned_addr learned my addr 172.21.15.77:0/1653412971 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:31.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.606+0000 7f0458fd2640 1 -- 172.21.15.77:0/1653412971 >> v2:172.21.15.121:3300/0 conn(0x7f0454143940 msgr2=0x7f045406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:31.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.606+0000 7f0458fd2640 1 --2- 172.21.15.77:0/1653412971 >> v2:172.21.15.121:3300/0 conn(0x7f0454143940 0x7f045406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.606+0000 7f0458fd2640 1 -- 172.21.15.77:0/1653412971 >> v1:172.21.15.121:6789/0 conn(0x7f045415ecf0 legacy=0x7f045415f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:31.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.606+0000 7f0458fd2640 1 -- 172.21.15.77:0/1653412971 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0454071700 con 0x7f0454143d20 2024-04-25T01:23:31.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.606+0000 7f0458fd2640 1 --2- 172.21.15.77:0/1653412971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0454143d20 0x7f045406eb90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0448009870 tx=0x7f044802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f7e856b3902703 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:31.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.607+0000 7f0453fff640 1 -- 172.21.15.77:0/1653412971 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448037070 con 0x7f0454143d20 2024-04-25T01:23:31.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.607+0000 7f0453fff640 1 -- 172.21.15.77:0/1653412971 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0448032440 con 0x7f0454143d20 2024-04-25T01:23:31.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.607+0000 7f0453fff640 1 -- 172.21.15.77:0/1653412971 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448032810 con 0x7f0454143d20 2024-04-25T01:23:31.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.607+0000 7f045ba37640 1 -- 172.21.15.77:0/1653412971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0454143d20 msgr2=0x7f045406eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.607+0000 7f045ba37640 1 --2- 172.21.15.77:0/1653412971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0454143d20 0x7f045406eb90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0448009870 tx=0x7f044802a270 comp rx=0 tx=0).stop 2024-04-25T01:23:31.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.607+0000 7f045ba37640 1 -- 172.21.15.77:0/1653412971 shutdown_connections 2024-04-25T01:23:31.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.607+0000 7f045ba37640 1 --2- 172.21.15.77:0/1653412971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0454143d20 0x7f045406eb90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.607+0000 7f045ba37640 1 --2- 172.21.15.77:0/1653412971 >> v2:172.21.15.121:3300/0 conn(0x7f0454143940 0x7f045406e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.607+0000 7f045ba37640 1 -- 172.21.15.77:0/1653412971 >> 172.21.15.77:0/1653412971 conn(0x7f04540b1820 msgr2=0x7f04540b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:31.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.607+0000 7f045ba37640 1 -- 172.21.15.77:0/1653412971 shutdown_connections 2024-04-25T01:23:31.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.607+0000 7f045ba37640 1 -- 172.21.15.77:0/1653412971 wait complete. 2024-04-25T01:23:31.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.608+0000 7f045ba37640 1 Processor -- start 2024-04-25T01:23:31.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.608+0000 7f045ba37640 1 -- start start 2024-04-25T01:23:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f045ba37640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0454143940 0x7f0454134cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f045ba37640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0454143d20 0x7f045415a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f045ba37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f045415ecf0 0x7f04541535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f045ba37640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0454071ff0 con 0x7f045415ecf0 2024-04-25T01:23:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f045ba37640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0454071e70 con 0x7f0454143d20 2024-04-25T01:23:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f045ba37640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0454072170 con 0x7f0454143940 2024-04-25T01:23:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f0459fd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f045415ecf0 0x7f04541535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f0459fd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f045415ecf0 0x7f04541535b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42386/0 (socket says 172.21.15.77:42386) 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f0459fd4640 1 -- 172.21.15.77:0/4071361677 learned_addr learned my addr 172.21.15.77:0/4071361677 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f0458fd2640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0454143d20 0x7f045415a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f04597d3640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0454143940 0x7f0454134cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f0459fd4640 1 -- 172.21.15.77:0/4071361677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0454143940 msgr2=0x7f0454134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f0459fd4640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0454143940 0x7f0454134cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f0459fd4640 1 -- 172.21.15.77:0/4071361677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0454143d20 msgr2=0x7f045415a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f0459fd4640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0454143d20 0x7f045415a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f0459fd4640 1 -- 172.21.15.77:0/4071361677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045415a720 con 0x7f045415ecf0 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.609+0000 7f04597d3640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0454143940 0x7f0454134cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.610+0000 7f0459fd4640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f045415ecf0 0x7f04541535b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f044400b490 tx=0x7f044400b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.610+0000 7f0451ffb640 1 -- 172.21.15.77:0/4071361677 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444015020 con 0x7f045415ecf0 2024-04-25T01:23:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.610+0000 7f0451ffb640 1 -- 172.21.15.77:0/4071361677 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0444011850 con 0x7f045415ecf0 2024-04-25T01:23:31.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.610+0000 7f045ba37640 1 -- 172.21.15.77:0/4071361677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0454157870 con 0x7f045415ecf0 2024-04-25T01:23:31.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.610+0000 7f0451ffb640 1 -- 172.21.15.77:0/4071361677 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444011c40 con 0x7f045415ecf0 2024-04-25T01:23:31.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.610+0000 7f045ba37640 1 -- 172.21.15.77:0/4071361677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04540d0720 con 0x7f045415ecf0 2024-04-25T01:23:31.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.611+0000 7f045ba37640 1 -- 172.21.15.77:0/4071361677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0454156120 con 0x7f045415ecf0 2024-04-25T01:23:31.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.611+0000 7f0451ffb640 1 -- 172.21.15.77:0/4071361677 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0444018020 con 0x7f045415ecf0 2024-04-25T01:23:31.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.612+0000 7f0451ffb640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f044006be30 0x7f044006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.612+0000 7f0451ffb640 1 -- 172.21.15.77:0/4071361677 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f044408aea0 con 0x7f045415ecf0 2024-04-25T01:23:31.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.612+0000 7f04597d3640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f044006be30 0x7f044006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.612+0000 7f04597d3640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f044006be30 0x7f044006e2f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f044c007eb0 tx=0x7f044c0073c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:31.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.616+0000 7f0451ffb640 1 -- 172.21.15.77:0/4071361677 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f044401f020 con 0x7f045415ecf0 2024-04-25T01:23:31.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.784+0000 7f045ba37640 1 -- 172.21.15.77:0/4071361677 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f04540d0960 con 0x7f044006be30 2024-04-25T01:23:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.789+0000 7f0451ffb640 1 -- 172.21.15.77:0/4071361677 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246638 (secure 0 0 0) 0x7f04540d0960 con 0x7f044006be30 2024-04-25T01:23:31.773 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:23:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.792+0000 7f045ba37640 1 -- 172.21.15.77:0/4071361677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f044006be30 msgr2=0x7f044006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.792+0000 7f045ba37640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f044006be30 0x7f044006e2f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f044c007eb0 tx=0x7f044c0073c0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.792+0000 7f045ba37640 1 -- 172.21.15.77:0/4071361677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f045415ecf0 msgr2=0x7f04541535b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.792+0000 7f045ba37640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f045415ecf0 0x7f04541535b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f044400b490 tx=0x7f044400b950 comp rx=0 tx=0).stop 2024-04-25T01:23:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.793+0000 7f045ba37640 1 -- 172.21.15.77:0/4071361677 shutdown_connections 2024-04-25T01:23:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.793+0000 7f045ba37640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f044006be30 0x7f044006e2f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.793+0000 7f045ba37640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f045415ecf0 0x7f04541535b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.793+0000 7f045ba37640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0454143d20 0x7f045415a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.793+0000 7f045ba37640 1 --2- 172.21.15.77:0/4071361677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0454143940 0x7f0454134cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.793+0000 7f045ba37640 1 -- 172.21.15.77:0/4071361677 >> 172.21.15.77:0/4071361677 conn(0x7f04540b1820 msgr2=0x7f0454072550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.793+0000 7f045ba37640 1 -- 172.21.15.77:0/4071361677 shutdown_connections 2024-04-25T01:23:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.793+0000 7f045ba37640 1 -- 172.21.15.77:0/4071361677 wait complete. 2024-04-25T01:23:31.784 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-04-25T01:23:31.176800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221792,"kb_used_data":5608,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752928,"statfs":{"total":773094113280,"available":772866998272,"internally_reserved":0,"allocated":5742592,"data_stored":3179920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.354926"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828699+0000","last_change":"2024-04-25T01:23:27.828699+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828699+0000","last_undegraded":"2024-04-25T01:23:27.828699+0000","last_fullsized":"2024-04-25T01:23:27.828699+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833069+0000","last_change":"2024-04-25T01:23:27.833069+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833069+0000","last_undegraded":"2024-04-25T01:23:27.833069+0000","last_fullsized":"2024-04-25T01:23:27.833069+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838229+0000","last_change":"2024-04-25T01:23:27.838229+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838229+0000","last_undegraded":"2024-04-25T01:23:27.838229+0000","last_fullsized":"2024-04-25T01:23:27.838229+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830113+0000","last_change":"2024-04-25T01:23:27.830113+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830113+0000","last_undegraded":"2024-04-25T01:23:27.830113+0000","last_fullsized":"2024-04-25T01:23:27.830113+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830687+0000","last_change":"2024-04-25T01:23:27.830687+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830687+0000","last_undegraded":"2024-04-25T01:23:27.830687+0000","last_fullsized":"2024-04-25T01:23:27.830687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828146+0000","last_change":"2024-04-25T01:23:27.828146+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828146+0000","last_undegraded":"2024-04-25T01:23:27.828146+0000","last_fullsized":"2024-04-25T01:23:27.828146+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838894+0000","last_change":"2024-04-25T01:23:27.838894+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838894+0000","last_undegraded":"2024-04-25T01:23:27.838894+0000","last_fullsized":"2024-04-25T01:23:27.838894+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834175+0000","last_change":"2024-04-25T01:23:27.834175+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834175+0000","last_undegraded":"2024-04-25T01:23:27.834175+0000","last_fullsized":"2024-04-25T01:23:27.834175+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836796+0000","last_change":"2024-04-25T01:23:27.836796+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836796+0000","last_undegraded":"2024-04-25T01:23:27.836796+0000","last_fullsized":"2024-04-25T01:23:27.836796+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827603+0000","last_change":"2024-04-25T01:23:27.827603+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827603+0000","last_undegraded":"2024-04-25T01:23:27.827603+0000","last_fullsized":"2024-04-25T01:23:27.827603+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830254+0000","last_change":"2024-04-25T01:23:27.830254+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830254+0000","last_undegraded":"2024-04-25T01:23:27.830254+0000","last_fullsized":"2024-04-25T01:23:27.830254+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833652+0000","last_change":"2024-04-25T01:23:27.833652+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833652+0000","last_undegraded":"2024-04-25T01:23:27.833652+0000","last_fullsized":"2024-04-25T01:23:27.833652+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827511+0000","last_change":"2024-04-25T01:23:27.827511+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827511+0000","last_undegraded":"2024-04-25T01:23:27.827511+0000","last_fullsized":"2024-04-25T01:23:27.827511+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838536+0000","last_change":"2024-04-25T01:23:27.838536+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838536+0000","last_undegraded":"2024-04-25T01:23:27.838536+0000","last_fullsized":"2024-04-25T01:23:27.838536+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.829534+0000","last_change":"2024-04-25T01:23:27.829534+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.829534+0000","last_undegraded":"2024-04-25T01:23:27.829534+0000","last_fullsized":"2024-04-25T01:23:27.829534+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.825966+0000","last_change":"2024-04-25T01:23:27.825966+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.825966+0000","last_undegraded":"2024-04-25T01:23:27.825966+0000","last_fullsized":"2024-04-25T01:23:27.825966+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838142+0000","last_change":"2024-04-25T01:23:27.838142+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838142+0000","last_undegraded":"2024-04-25T01:23:27.838142+0000","last_fullsized":"2024-04-25T01:23:27.838142+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832007+0000","last_change":"2024-04-25T01:23:27.832007+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832007+0000","last_undegraded":"2024-04-25T01:23:27.832007+0000","last_fullsized":"2024-04-25T01:23:27.832007+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836983+0000","last_change":"2024-04-25T01:23:27.836983+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836983+0000","last_undegraded":"2024-04-25T01:23:27.836983+0000","last_fullsized":"2024-04-25T01:23:27.836983+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832039+0000","last_change":"2024-04-25T01:23:27.832039+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832039+0000","last_undegraded":"2024-04-25T01:23:27.832039+0000","last_fullsized":"2024-04-25T01:23:27.832039+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830258+0000","last_change":"2024-04-25T01:23:27.830258+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830258+0000","last_undegraded":"2024-04-25T01:23:27.830258+0000","last_fullsized":"2024-04-25T01:23:27.830258+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833623+0000","last_change":"2024-04-25T01:23:27.833623+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833623+0000","last_undegraded":"2024-04-25T01:23:27.833623+0000","last_fullsized":"2024-04-25T01:23:27.833623+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830382+0000","last_change":"2024-04-25T01:23:27.830382+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830382+0000","last_undegraded":"2024-04-25T01:23:27.830382+0000","last_fullsized":"2024-04-25T01:23:27.830382+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836595+0000","last_change":"2024-04-25T01:23:27.836595+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836595+0000","last_undegraded":"2024-04-25T01:23:27.836595+0000","last_fullsized":"2024-04-25T01:23:27.836595+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836402+0000","last_change":"2024-04-25T01:23:27.836402+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836402+0000","last_undegraded":"2024-04-25T01:23:27.836402+0000","last_fullsized":"2024-04-25T01:23:27.836402+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835991+0000","last_change":"2024-04-25T01:23:27.835991+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835991+0000","last_undegraded":"2024-04-25T01:23:27.835991+0000","last_fullsized":"2024-04-25T01:23:27.835991+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827570+0000","last_change":"2024-04-25T01:23:27.827570+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827570+0000","last_undegraded":"2024-04-25T01:23:27.827570+0000","last_fullsized":"2024-04-25T01:23:27.827570+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831597+0000","last_change":"2024-04-25T01:23:27.831597+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831597+0000","last_undegraded":"2024-04-25T01:23:27.831597+0000","last_fullsized":"2024-04-25T01:23:27.831597+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836573+0000","last_change":"2024-04-25T01:23:27.836573+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836573+0000","last_undegraded":"2024-04-25T01:23:27.836573+0000","last_fullsized":"2024-04-25T01:23:27.836573+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830474+0000","last_change":"2024-04-25T01:23:27.830474+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830474+0000","last_undegraded":"2024-04-25T01:23:27.830474+0000","last_fullsized":"2024-04-25T01:23:27.830474+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835732+0000","last_change":"2024-04-25T01:23:27.835732+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835732+0000","last_undegraded":"2024-04-25T01:23:27.835732+0000","last_fullsized":"2024-04-25T01:23:27.835732+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828172+0000","last_change":"2024-04-25T01:23:27.828172+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828172+0000","last_undegraded":"2024-04-25T01:23:27.828172+0000","last_fullsized":"2024-04-25T01:23:27.828172+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830914+0000","last_change":"2024-04-25T01:23:27.830914+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830914+0000","last_undegraded":"2024-04-25T01:23:27.830914+0000","last_fullsized":"2024-04-25T01:23:27.830914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844780+0000","last_change":"2024-04-25T01:23:27.840369+0000","last_active":"2024-04-25T01:23:27.844780+0000","last_peered":"2024-04-25T01:23:27.844780+0000","last_clean":"2024-04-25T01:23:27.844780+0000","last_became_active":"2024-04-25T01:23:27.839987+0000","last_became_peered":"2024-04-25T01:23:27.839987+0000","last_unstale":"2024-04-25T01:23:27.844780+0000","last_undegraded":"2024-04-25T01:23:27.844780+0000","last_fullsized":"2024-04-25T01:23:27.844780+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.219905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826648+0000","last_change":"2024-04-25T01:23:27.826648+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826648+0000","last_undegraded":"2024-04-25T01:23:27.826648+0000","last_fullsized":"2024-04-25T01:23:27.826648+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.841990+0000","last_change":"2024-04-25T01:23:27.838394+0000","last_active":"2024-04-25T01:23:27.841990+0000","last_peered":"2024-04-25T01:23:27.841990+0000","last_clean":"2024-04-25T01:23:27.841990+0000","last_became_active":"2024-04-25T01:23:27.838114+0000","last_became_peered":"2024-04-25T01:23:27.838114+0000","last_unstale":"2024-04-25T01:23:27.841990+0000","last_undegraded":"2024-04-25T01:23:27.841990+0000","last_fullsized":"2024-04-25T01:23:27.841990+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:48.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832822+0000","last_change":"2024-04-25T01:23:27.832822+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832822+0000","last_undegraded":"2024-04-25T01:23:27.832822+0000","last_fullsized":"2024-04-25T01:23:27.832822+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.842100+0000","last_change":"2024-04-25T01:23:27.839860+0000","last_active":"2024-04-25T01:23:27.842100+0000","last_peered":"2024-04-25T01:23:27.842100+0000","last_clean":"2024-04-25T01:23:27.842100+0000","last_became_active":"2024-04-25T01:23:27.839589+0000","last_became_peered":"2024-04-25T01:23:27.839589+0000","last_unstale":"2024-04-25T01:23:27.842100+0000","last_undegraded":"2024-04-25T01:23:27.842100+0000","last_fullsized":"2024-04-25T01:23:27.842100+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:52.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835561+0000","last_change":"2024-04-25T01:23:27.835561+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835561+0000","last_undegraded":"2024-04-25T01:23:27.835561+0000","last_fullsized":"2024-04-25T01:23:27.835561+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847649+0000","last_change":"2024-04-25T01:23:27.842142+0000","last_active":"2024-04-25T01:23:27.847649+0000","last_peered":"2024-04-25T01:23:27.847649+0000","last_clean":"2024-04-25T01:23:27.847649+0000","last_became_active":"2024-04-25T01:23:27.841740+0000","last_became_peered":"2024-04-25T01:23:27.841740+0000","last_unstale":"2024-04-25T01:23:27.847649+0000","last_undegraded":"2024-04-25T01:23:27.847649+0000","last_fullsized":"2024-04-25T01:23:27.847649+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:13.292186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831025+0000","last_change":"2024-04-25T01:23:27.831025+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831025+0000","last_undegraded":"2024-04-25T01:23:27.831025+0000","last_fullsized":"2024-04-25T01:23:27.831025+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844533+0000","last_change":"2024-04-25T01:23:27.831214+0000","last_active":"2024-04-25T01:23:27.844533+0000","last_peered":"2024-04-25T01:23:27.844533+0000","last_clean":"2024-04-25T01:23:27.844533+0000","last_became_active":"2024-04-25T01:23:27.830883+0000","last_became_peered":"2024-04-25T01:23:27.830883+0000","last_unstale":"2024-04-25T01:23:27.844533+0000","last_undegraded":"2024-04-25T01:23:27.844533+0000","last_fullsized":"2024-04-25T01:23:27.844533+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:40.061850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830940+0000","last_change":"2024-04-25T01:23:27.830940+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830940+0000","last_undegraded":"2024-04-25T01:23:27.830940+0000","last_fullsized":"2024-04-25T01:23:27.830940+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844655+0000","last_change":"2024-04-25T01:23:27.839506+0000","last_active":"2024-04-25T01:23:27.844655+0000","last_peered":"2024-04-25T01:23:27.844655+0000","last_clean":"2024-04-25T01:23:27.844655+0000","last_became_active":"2024-04-25T01:23:27.838600+0000","last_became_peered":"2024-04-25T01:23:27.838600+0000","last_unstale":"2024-04-25T01:23:27.844655+0000","last_undegraded":"2024-04-25T01:23:27.844655+0000","last_fullsized":"2024-04-25T01:23:27.844655+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:48.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833846+0000","last_change":"2024-04-25T01:23:27.833846+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833846+0000","last_undegraded":"2024-04-25T01:23:27.833846+0000","last_fullsized":"2024-04-25T01:23:27.833846+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847468+0000","last_change":"2024-04-25T01:23:27.843276+0000","last_active":"2024-04-25T01:23:27.847468+0000","last_peered":"2024-04-25T01:23:27.847468+0000","last_clean":"2024-04-25T01:23:27.847468+0000","last_became_active":"2024-04-25T01:23:27.842824+0000","last_became_peered":"2024-04-25T01:23:27.842824+0000","last_unstale":"2024-04-25T01:23:27.847468+0000","last_undegraded":"2024-04-25T01:23:27.847468+0000","last_fullsized":"2024-04-25T01:23:27.847468+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:48.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831669+0000","last_change":"2024-04-25T01:23:27.831669+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831669+0000","last_undegraded":"2024-04-25T01:23:27.831669+0000","last_fullsized":"2024-04-25T01:23:27.831669+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848229+0000","last_change":"2024-04-25T01:23:27.842530+0000","last_active":"2024-04-25T01:23:27.848229+0000","last_peered":"2024-04-25T01:23:27.848229+0000","last_clean":"2024-04-25T01:23:27.848229+0000","last_became_active":"2024-04-25T01:23:27.842265+0000","last_became_peered":"2024-04-25T01:23:27.842265+0000","last_unstale":"2024-04-25T01:23:27.848229+0000","last_undegraded":"2024-04-25T01:23:27.848229+0000","last_fullsized":"2024-04-25T01:23:27.848229+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:49:28.578991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.829914+0000","last_change":"2024-04-25T01:23:27.829914+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.829914+0000","last_undegraded":"2024-04-25T01:23:27.829914+0000","last_fullsized":"2024-04-25T01:23:27.829914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847439+0000","last_change":"2024-04-25T01:23:27.842714+0000","last_active":"2024-04-25T01:23:27.847439+0000","last_peered":"2024-04-25T01:23:27.847439+0000","last_clean":"2024-04-25T01:23:27.847439+0000","last_became_active":"2024-04-25T01:23:27.842268+0000","last_became_peered":"2024-04-25T01:23:27.842268+0000","last_unstale":"2024-04-25T01:23:27.847439+0000","last_undegraded":"2024-04-25T01:23:27.847439+0000","last_fullsized":"2024-04-25T01:23:27.847439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:01.477667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827321+0000","last_change":"2024-04-25T01:23:27.827321+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827321+0000","last_undegraded":"2024-04-25T01:23:27.827321+0000","last_fullsized":"2024-04-25T01:23:27.827321+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844619+0000","last_change":"2024-04-25T01:23:27.839254+0000","last_active":"2024-04-25T01:23:27.844619+0000","last_peered":"2024-04-25T01:23:27.844619+0000","last_clean":"2024-04-25T01:23:27.844619+0000","last_became_active":"2024-04-25T01:23:27.838972+0000","last_became_peered":"2024-04-25T01:23:27.838972+0000","last_unstale":"2024-04-25T01:23:27.844619+0000","last_undegraded":"2024-04-25T01:23:27.844619+0000","last_fullsized":"2024-04-25T01:23:27.844619+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:04.719084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834497+0000","last_change":"2024-04-25T01:23:27.834497+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834497+0000","last_undegraded":"2024-04-25T01:23:27.834497+0000","last_fullsized":"2024-04-25T01:23:27.834497+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847498+0000","last_change":"2024-04-25T01:23:27.843358+0000","last_active":"2024-04-25T01:23:27.847498+0000","last_peered":"2024-04-25T01:23:27.847498+0000","last_clean":"2024-04-25T01:23:27.847498+0000","last_became_active":"2024-04-25T01:23:27.843098+0000","last_became_peered":"2024-04-25T01:23:27.843098+0000","last_unstale":"2024-04-25T01:23:27.847498+0000","last_undegraded":"2024-04-25T01:23:27.847498+0000","last_fullsized":"2024-04-25T01:23:27.847498+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:38.211801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830273+0000","last_change":"2024-04-25T01:23:27.830273+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830273+0000","last_undegraded":"2024-04-25T01:23:27.830273+0000","last_fullsized":"2024-04-25T01:23:27.830273+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846301+0000","last_change":"2024-04-25T01:23:27.841668+0000","last_active":"2024-04-25T01:23:27.846301+0000","last_peered":"2024-04-25T01:23:27.846301+0000","last_clean":"2024-04-25T01:23:27.846301+0000","last_became_active":"2024-04-25T01:23:27.841382+0000","last_became_peered":"2024-04-25T01:23:27.841382+0000","last_unstale":"2024-04-25T01:23:27.846301+0000","last_undegraded":"2024-04-25T01:23:27.846301+0000","last_fullsized":"2024-04-25T01:23:27.846301+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:00:11.736192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827805+0000","last_change":"2024-04-25T01:23:27.827805+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827805+0000","last_undegraded":"2024-04-25T01:23:27.827805+0000","last_fullsized":"2024-04-25T01:23:27.827805+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848483+0000","last_change":"2024-04-25T01:23:27.845237+0000","last_active":"2024-04-25T01:23:27.848483+0000","last_peered":"2024-04-25T01:23:27.848483+0000","last_clean":"2024-04-25T01:23:27.848483+0000","last_became_active":"2024-04-25T01:23:27.844336+0000","last_became_peered":"2024-04-25T01:23:27.844336+0000","last_unstale":"2024-04-25T01:23:27.848483+0000","last_undegraded":"2024-04-25T01:23:27.848483+0000","last_fullsized":"2024-04-25T01:23:27.848483+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:52.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834532+0000","last_change":"2024-04-25T01:23:27.834532+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834532+0000","last_undegraded":"2024-04-25T01:23:27.834532+0000","last_fullsized":"2024-04-25T01:23:27.834532+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.849509+0000","last_change":"2024-04-25T01:23:27.848962+0000","last_active":"2024-04-25T01:23:27.849509+0000","last_peered":"2024-04-25T01:23:27.849509+0000","last_clean":"2024-04-25T01:23:27.849509+0000","last_became_active":"2024-04-25T01:23:27.848694+0000","last_became_peered":"2024-04-25T01:23:27.848694+0000","last_unstale":"2024-04-25T01:23:27.849509+0000","last_undegraded":"2024-04-25T01:23:27.849509+0000","last_fullsized":"2024-04-25T01:23:27.849509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:04.282731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834242+0000","last_change":"2024-04-25T01:23:27.834242+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834242+0000","last_undegraded":"2024-04-25T01:23:27.834242+0000","last_fullsized":"2024-04-25T01:23:27.834242+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847532+0000","last_change":"2024-04-25T01:23:27.843254+0000","last_active":"2024-04-25T01:23:27.847532+0000","last_peered":"2024-04-25T01:23:27.847532+0000","last_clean":"2024-04-25T01:23:27.847532+0000","last_became_active":"2024-04-25T01:23:27.842506+0000","last_became_peered":"2024-04-25T01:23:27.842506+0000","last_unstale":"2024-04-25T01:23:27.847532+0000","last_undegraded":"2024-04-25T01:23:27.847532+0000","last_fullsized":"2024-04-25T01:23:27.847532+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:42.248852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835026+0000","last_change":"2024-04-25T01:23:27.835026+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835026+0000","last_undegraded":"2024-04-25T01:23:27.835026+0000","last_fullsized":"2024-04-25T01:23:27.835026+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846277+0000","last_change":"2024-04-25T01:23:27.841058+0000","last_active":"2024-04-25T01:23:27.846277+0000","last_peered":"2024-04-25T01:23:27.846277+0000","last_clean":"2024-04-25T01:23:27.846277+0000","last_became_active":"2024-04-25T01:23:27.840808+0000","last_became_peered":"2024-04-25T01:23:27.840808+0000","last_unstale":"2024-04-25T01:23:27.846277+0000","last_undegraded":"2024-04-25T01:23:27.846277+0000","last_fullsized":"2024-04-25T01:23:27.846277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:03.437919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833035+0000","last_change":"2024-04-25T01:23:27.833035+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833035+0000","last_undegraded":"2024-04-25T01:23:27.833035+0000","last_fullsized":"2024-04-25T01:23:27.833035+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848509+0000","last_change":"2024-04-25T01:23:27.846019+0000","last_active":"2024-04-25T01:23:27.848509+0000","last_peered":"2024-04-25T01:23:27.848509+0000","last_clean":"2024-04-25T01:23:27.848509+0000","last_became_active":"2024-04-25T01:23:27.845056+0000","last_became_peered":"2024-04-25T01:23:27.845056+0000","last_unstale":"2024-04-25T01:23:27.848509+0000","last_undegraded":"2024-04-25T01:23:27.848509+0000","last_fullsized":"2024-04-25T01:23:27.848509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:08.739387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826903+0000","last_change":"2024-04-25T01:23:27.826903+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826903+0000","last_undegraded":"2024-04-25T01:23:27.826903+0000","last_fullsized":"2024-04-25T01:23:27.826903+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.845756+0000","last_change":"2024-04-25T01:23:27.839989+0000","last_active":"2024-04-25T01:23:27.845756+0000","last_peered":"2024-04-25T01:23:27.845756+0000","last_clean":"2024-04-25T01:23:27.845756+0000","last_became_active":"2024-04-25T01:23:27.839737+0000","last_became_peered":"2024-04-25T01:23:27.839737+0000","last_unstale":"2024-04-25T01:23:27.845756+0000","last_undegraded":"2024-04-25T01:23:27.845756+0000","last_fullsized":"2024-04-25T01:23:27.845756+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:17.100820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830591+0000","last_change":"2024-04-25T01:23:27.830591+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830591+0000","last_undegraded":"2024-04-25T01:23:27.830591+0000","last_fullsized":"2024-04-25T01:23:27.830591+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.845062+0000","last_change":"2024-04-25T01:23:27.838563+0000","last_active":"2024-04-25T01:23:27.845062+0000","last_peered":"2024-04-25T01:23:27.845062+0000","last_clean":"2024-04-25T01:23:27.845062+0000","last_became_active":"2024-04-25T01:23:27.837764+0000","last_became_peered":"2024-04-25T01:23:27.837764+0000","last_unstale":"2024-04-25T01:23:27.845062+0000","last_undegraded":"2024-04-25T01:23:27.845062+0000","last_fullsized":"2024-04-25T01:23:27.845062+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:26.874008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826397+0000","last_change":"2024-04-25T01:23:27.826397+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826397+0000","last_undegraded":"2024-04-25T01:23:27.826397+0000","last_fullsized":"2024-04-25T01:23:27.826397+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847457+0000","last_change":"2024-04-25T01:23:27.843101+0000","last_active":"2024-04-25T01:23:27.847457+0000","last_peered":"2024-04-25T01:23:27.847457+0000","last_clean":"2024-04-25T01:23:27.847457+0000","last_became_active":"2024-04-25T01:23:27.842810+0000","last_became_peered":"2024-04-25T01:23:27.842810+0000","last_unstale":"2024-04-25T01:23:27.847457+0000","last_undegraded":"2024-04-25T01:23:27.847457+0000","last_fullsized":"2024-04-25T01:23:27.847457+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:04.719084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826783+0000","last_change":"2024-04-25T01:23:27.826783+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826783+0000","last_undegraded":"2024-04-25T01:23:27.826783+0000","last_fullsized":"2024-04-25T01:23:27.826783+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.838395+0000","last_change":"2024-04-25T01:23:27.834285+0000","last_active":"2024-04-25T01:23:27.838395+0000","last_peered":"2024-04-25T01:23:27.838395+0000","last_clean":"2024-04-25T01:23:27.838395+0000","last_became_active":"2024-04-25T01:23:27.834030+0000","last_became_peered":"2024-04-25T01:23:27.834030+0000","last_unstale":"2024-04-25T01:23:27.838395+0000","last_undegraded":"2024-04-25T01:23:27.838395+0000","last_fullsized":"2024-04-25T01:23:27.838395+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:33.027603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832642+0000","last_change":"2024-04-25T01:23:27.832642+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832642+0000","last_undegraded":"2024-04-25T01:23:27.832642+0000","last_fullsized":"2024-04-25T01:23:27.832642+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846225+0000","last_change":"2024-04-25T01:23:27.839744+0000","last_active":"2024-04-25T01:23:27.846225+0000","last_peered":"2024-04-25T01:23:27.846225+0000","last_clean":"2024-04-25T01:23:27.846225+0000","last_became_active":"2024-04-25T01:23:27.839478+0000","last_became_peered":"2024-04-25T01:23:27.839478+0000","last_unstale":"2024-04-25T01:23:27.846225+0000","last_undegraded":"2024-04-25T01:23:27.846225+0000","last_fullsized":"2024-04-25T01:23:27.846225+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:44.264978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833162+0000","last_change":"2024-04-25T01:23:27.833162+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833162+0000","last_undegraded":"2024-04-25T01:23:27.833162+0000","last_fullsized":"2024-04-25T01:23:27.833162+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848293+0000","last_change":"2024-04-25T01:23:27.843276+0000","last_active":"2024-04-25T01:23:27.848293+0000","last_peered":"2024-04-25T01:23:27.848293+0000","last_clean":"2024-04-25T01:23:27.848293+0000","last_became_active":"2024-04-25T01:23:27.842913+0000","last_became_peered":"2024-04-25T01:23:27.842913+0000","last_unstale":"2024-04-25T01:23:27.848293+0000","last_undegraded":"2024-04-25T01:23:27.848293+0000","last_fullsized":"2024-04-25T01:23:27.848293+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:33.027603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826481+0000","last_change":"2024-04-25T01:23:27.826481+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826481+0000","last_undegraded":"2024-04-25T01:23:27.826481+0000","last_fullsized":"2024-04-25T01:23:27.826481+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846474+0000","last_change":"2024-04-25T01:23:27.843723+0000","last_active":"2024-04-25T01:23:27.846474+0000","last_peered":"2024-04-25T01:23:27.846474+0000","last_clean":"2024-04-25T01:23:27.846474+0000","last_became_active":"2024-04-25T01:23:27.843464+0000","last_became_peered":"2024-04-25T01:23:27.843464+0000","last_unstale":"2024-04-25T01:23:27.846474+0000","last_undegraded":"2024-04-25T01:23:27.846474+0000","last_fullsized":"2024-04-25T01:23:27.846474+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:01.926860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826655+0000","last_change":"2024-04-25T01:23:27.826655+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826655+0000","last_undegraded":"2024-04-25T01:23:27.826655+0000","last_fullsized":"2024-04-25T01:23:27.826655+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848098+0000","last_change":"2024-04-25T01:23:27.847328+0000","last_active":"2024-04-25T01:23:27.848098+0000","last_peered":"2024-04-25T01:23:27.848098+0000","last_clean":"2024-04-25T01:23:27.848098+0000","last_became_active":"2024-04-25T01:23:27.847046+0000","last_became_peered":"2024-04-25T01:23:27.847046+0000","last_unstale":"2024-04-25T01:23:27.848098+0000","last_undegraded":"2024-04-25T01:23:27.848098+0000","last_fullsized":"2024-04-25T01:23:27.848098+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:59.145724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827287+0000","last_change":"2024-04-25T01:23:27.827287+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827287+0000","last_undegraded":"2024-04-25T01:23:27.827287+0000","last_fullsized":"2024-04-25T01:23:27.827287+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847476+0000","last_change":"2024-04-25T01:23:27.842809+0000","last_active":"2024-04-25T01:23:27.847476+0000","last_peered":"2024-04-25T01:23:27.847476+0000","last_clean":"2024-04-25T01:23:27.847476+0000","last_became_active":"2024-04-25T01:23:27.842561+0000","last_became_peered":"2024-04-25T01:23:27.842561+0000","last_unstale":"2024-04-25T01:23:27.847476+0000","last_undegraded":"2024-04-25T01:23:27.847476+0000","last_fullsized":"2024-04-25T01:23:27.847476+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:28:34.927620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835126+0000","last_change":"2024-04-25T01:23:27.835126+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835126+0000","last_undegraded":"2024-04-25T01:23:27.835126+0000","last_fullsized":"2024-04-25T01:23:27.835126+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848367+0000","last_change":"2024-04-25T01:23:27.844808+0000","last_active":"2024-04-25T01:23:27.848367+0000","last_peered":"2024-04-25T01:23:27.848367+0000","last_clean":"2024-04-25T01:23:27.848367+0000","last_became_active":"2024-04-25T01:23:27.844541+0000","last_became_peered":"2024-04-25T01:23:27.844541+0000","last_unstale":"2024-04-25T01:23:27.848367+0000","last_undegraded":"2024-04-25T01:23:27.848367+0000","last_fullsized":"2024-04-25T01:23:27.848367+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:48.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832949+0000","last_change":"2024-04-25T01:23:27.832949+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832949+0000","last_undegraded":"2024-04-25T01:23:27.832949+0000","last_fullsized":"2024-04-25T01:23:27.832949+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846394+0000","last_change":"2024-04-25T01:23:27.842454+0000","last_active":"2024-04-25T01:23:27.846394+0000","last_peered":"2024-04-25T01:23:27.846394+0000","last_clean":"2024-04-25T01:23:27.846394+0000","last_became_active":"2024-04-25T01:23:27.842209+0000","last_became_peered":"2024-04-25T01:23:27.842209+0000","last_unstale":"2024-04-25T01:23:27.846394+0000","last_undegraded":"2024-04-25T01:23:27.846394+0000","last_fullsized":"2024-04-25T01:23:27.846394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:22.705548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826687+0000","last_change":"2024-04-25T01:23:27.826687+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826687+0000","last_undegraded":"2024-04-25T01:23:27.826687+0000","last_fullsized":"2024-04-25T01:23:27.826687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846248+0000","last_change":"2024-04-25T01:23:27.841080+0000","last_active":"2024-04-25T01:23:27.846248+0000","last_peered":"2024-04-25T01:23:27.846248+0000","last_clean":"2024-04-25T01:23:27.846248+0000","last_became_active":"2024-04-25T01:23:27.840813+0000","last_became_peered":"2024-04-25T01:23:27.840813+0000","last_unstale":"2024-04-25T01:23:27.846248+0000","last_undegraded":"2024-04-25T01:23:27.846248+0000","last_fullsized":"2024-04-25T01:23:27.846248+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:44.720263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833838+0000","last_change":"2024-04-25T01:23:27.833838+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833838+0000","last_undegraded":"2024-04-25T01:23:27.833838+0000","last_fullsized":"2024-04-25T01:23:27.833838+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844601+0000","last_change":"2024-04-25T01:23:27.841167+0000","last_active":"2024-04-25T01:23:27.844601+0000","last_peered":"2024-04-25T01:23:27.844601+0000","last_clean":"2024-04-25T01:23:27.844601+0000","last_became_active":"2024-04-25T01:23:27.840932+0000","last_became_peered":"2024-04-25T01:23:27.840932+0000","last_unstale":"2024-04-25T01:23:27.844601+0000","last_undegraded":"2024-04-25T01:23:27.844601+0000","last_fullsized":"2024-04-25T01:23:27.844601+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:59.145724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836435+0000","last_change":"2024-04-25T01:23:27.836435+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836435+0000","last_undegraded":"2024-04-25T01:23:27.836435+0000","last_fullsized":"2024-04-25T01:23:27.836435+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.842097+0000","last_change":"2024-04-25T01:23:27.838507+0000","last_active":"2024-04-25T01:23:27.842097+0000","last_peered":"2024-04-25T01:23:27.842097+0000","last_clean":"2024-04-25T01:23:27.842097+0000","last_became_active":"2024-04-25T01:23:27.838016+0000","last_became_peered":"2024-04-25T01:23:27.838016+0000","last_unstale":"2024-04-25T01:23:27.842097+0000","last_undegraded":"2024-04-25T01:23:27.842097+0000","last_fullsized":"2024-04-25T01:23:27.842097+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:09.155513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831341+0000","last_change":"2024-04-25T01:23:27.831341+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831341+0000","last_undegraded":"2024-04-25T01:23:27.831341+0000","last_fullsized":"2024-04-25T01:23:27.831341+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846638+0000","last_change":"2024-04-25T01:23:27.841576+0000","last_active":"2024-04-25T01:23:27.846638+0000","last_peered":"2024-04-25T01:23:27.846638+0000","last_clean":"2024-04-25T01:23:27.846638+0000","last_became_active":"2024-04-25T01:23:27.839038+0000","last_became_peered":"2024-04-25T01:23:27.839038+0000","last_unstale":"2024-04-25T01:23:27.846638+0000","last_undegraded":"2024-04-25T01:23:27.846638+0000","last_fullsized":"2024-04-25T01:23:27.846638+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:26:54.800425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":222,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.833349+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:23:27.833349+0000","last_peered":"2024-04-25T01:23:27.833349+0000","last_clean":"2024-04-25T01:23:27.833349+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:23:27.833349+0000","last_undegraded":"2024-04-25T01:23:27.833349+0000","last_fullsized":"2024-04-25T01:23:27.833349+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1163264,"data_stored":840266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1163264,"data_stored":840266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:23:31.788 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:23:31.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.941+0000 7fb4af838640 1 Processor -- start 2024-04-25T01:23:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.941+0000 7fb4af838640 1 -- start start 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.942+0000 7fb4af838640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb4a80cfaf0 0x7fb4a80d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.942+0000 7fb4af838640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4a80d1e90 con 0x7fb4a80d4a10 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.942+0000 7fb4af838640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4a80b54a0 con 0x7fb4a815e4c0 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.942+0000 7fb4af838640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fb4a80d2130 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.942+0000 7fb4acdd3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb4a80d4a10 0x7fb4a80b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36876/0 (socket says 172.21.15.77:36876) 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.942+0000 7fb4acdd3640 1 -- 172.21.15.77:0/1408647945 learned_addr learned my addr 172.21.15.77:0/1408647945 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.942+0000 7fb4addd5640 1 --2- 172.21.15.77:0/1408647945 >> v2:172.21.15.77:3300/0 conn(0x7fb4a80cfaf0 0x7fb4a80d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.943+0000 7fb49ffff640 1 -- 172.21.15.77:0/1408647945 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4015069940 0 0) 0x7fb4a80d1e90 con 0x7fb4a80d4a10 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.943+0000 7fb49ffff640 1 -- 172.21.15.77:0/1408647945 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb494003520 con 0x7fb4a80d4a10 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.943+0000 7fb4addd5640 1 -- 172.21.15.77:0/1408647945 >> v1:172.21.15.121:6789/0 conn(0x7fb4a815e4c0 legacy=0x7fb4a80d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.943+0000 7fb4addd5640 1 -- 172.21.15.77:0/1408647945 >> v1:172.21.15.77:6789/0 conn(0x7fb4a80d4a10 legacy=0x7fb4a80b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.943+0000 7fb4addd5640 1 -- 172.21.15.77:0/1408647945 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a80b5780 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.943+0000 7fb4addd5640 1 --2- 172.21.15.77:0/1408647945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80cfaf0 0x7fb4a80d0900 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00097a0 tx=0x7fb4a002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec0ccbc8c8046d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.943+0000 7fb49ffff640 1 -- 172.21.15.77:0/1408647945 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0037070 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.943+0000 7fb49ffff640 1 -- 172.21.15.77:0/1408647945 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4a0032440 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.944+0000 7fb49ffff640 1 -- 172.21.15.77:0/1408647945 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0032810 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.944+0000 7fb4af838640 1 -- 172.21.15.77:0/1408647945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80cfaf0 msgr2=0x7fb4a80d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.944+0000 7fb4af838640 1 --2- 172.21.15.77:0/1408647945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80cfaf0 0x7fb4a80d0900 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00097a0 tx=0x7fb4a002a1a0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.944+0000 7fb4af838640 1 -- 172.21.15.77:0/1408647945 shutdown_connections 2024-04-25T01:23:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.944+0000 7fb4af838640 1 --2- 172.21.15.77:0/1408647945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80cfaf0 0x7fb4a80d0900 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.944+0000 7fb4af838640 1 -- 172.21.15.77:0/1408647945 >> 172.21.15.77:0/1408647945 conn(0x7fb4a80d3ef0 msgr2=0x7fb4a80d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.944+0000 7fb4af838640 1 -- 172.21.15.77:0/1408647945 shutdown_connections 2024-04-25T01:23:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.944+0000 7fb4af838640 1 -- 172.21.15.77:0/1408647945 wait complete. 2024-04-25T01:23:31.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.945+0000 7fb4af838640 1 Processor -- start 2024-04-25T01:23:31.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.945+0000 7fb4af838640 1 -- start start 2024-04-25T01:23:31.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4af838640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80cfaf0 0x7fb4a80c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4af838640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4a80d4a10 0x7fb4a8156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4af838640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a815e4c0 0x7fb4a80cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4af838640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a80b30a0 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4af838640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a80b2f20 con 0x7fb4a815e4c0 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4af838640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb4a815e8a0 con 0x7fb4a80d4a10 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4ad5d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80cfaf0 0x7fb4a80c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4ad5d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80cfaf0 0x7fb4a80c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42396/0 (socket says 172.21.15.77:42396) 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4ad5d4640 1 -- 172.21.15.77:0/1161033915 learned_addr learned my addr 172.21.15.77:0/1161033915 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4ad5d4640 1 -- 172.21.15.77:0/1161033915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4a80d4a10 msgr2=0x7fb4a8156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4ad5d4640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4a80d4a10 0x7fb4a8156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4ad5d4640 1 -- 172.21.15.77:0/1161033915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a815e4c0 msgr2=0x7fb4a80cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4ad5d4640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a815e4c0 0x7fb4a80cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4ad5d4640 1 -- 172.21.15.77:0/1161033915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a8156960 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb4ad5d4640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80cfaf0 0x7fb4a80c8fc0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb4a400c7c0 tx=0x7fb4a400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.946+0000 7fb49dffb640 1 -- 172.21.15.77:0/1161033915 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4016020 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.947+0000 7fb49dffb640 1 -- 172.21.15.77:0/1161033915 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4a4006d50 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.947+0000 7fb4af838640 1 -- 172.21.15.77:0/1161033915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a80c7f20 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.947+0000 7fb49dffb640 1 -- 172.21.15.77:0/1161033915 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4003bb0 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.947+0000 7fb4af838640 1 -- 172.21.15.77:0/1161033915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a80c84e0 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.948+0000 7fb4af838640 1 -- 172.21.15.77:0/1161033915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4a8068fb0 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.948+0000 7fb49dffb640 1 -- 172.21.15.77:0/1161033915 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb4a400a850 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.949+0000 7fb49dffb640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb48c06be30 0x7fb48c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.949+0000 7fb49dffb640 1 -- 172.21.15.77:0/1161033915 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fb4a408ace0 con 0x7fb4a80cfaf0 2024-04-25T01:23:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.949+0000 7fb4acdd3640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb48c06be30 0x7fb48c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.949+0000 7fb4acdd3640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb48c06be30 0x7fb48c06e2f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80b9470 tx=0x7fb498007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:31.953+0000 7fb49dffb640 1 -- 172.21.15.77:0/1161033915 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb4a4020020 con 0x7fb4a80cfaf0 2024-04-25T01:23:32.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.122+0000 7fb4af838640 1 -- 172.21.15.77:0/1161033915 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb4a80b4e60 con 0x7fb48c06be30 2024-04-25T01:23:32.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.128+0000 7fb49dffb640 1 -- 172.21.15.77:0/1161033915 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246638 (secure 0 0 0) 0x7fb4a80b4e60 con 0x7fb48c06be30 2024-04-25T01:23:32.113 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:32.114 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:23:32.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 -- 172.21.15.77:0/1161033915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb48c06be30 msgr2=0x7fb48c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb48c06be30 0x7fb48c06e2f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80b9470 tx=0x7fb498007930 comp rx=0 tx=0).stop 2024-04-25T01:23:32.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 -- 172.21.15.77:0/1161033915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80cfaf0 msgr2=0x7fb4a80c8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80cfaf0 0x7fb4a80c8fc0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb4a400c7c0 tx=0x7fb4a400cc80 comp rx=0 tx=0).stop 2024-04-25T01:23:32.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 -- 172.21.15.77:0/1161033915 shutdown_connections 2024-04-25T01:23:32.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb48c06be30 0x7fb48c06e2f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a815e4c0 0x7fb4a80cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4a80d4a10 0x7fb4a8156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 --2- 172.21.15.77:0/1161033915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80cfaf0 0x7fb4a80c8fc0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 -- 172.21.15.77:0/1161033915 >> 172.21.15.77:0/1161033915 conn(0x7fb4a80d3ef0 msgr2=0x7fb4a80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:32.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 -- 172.21.15.77:0/1161033915 shutdown_connections 2024-04-25T01:23:32.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.131+0000 7fb4af838640 1 -- 172.21.15.77:0/1161033915 wait complete. 2024-04-25T01:23:32.123 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-04-25T01:23:31.176800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221792,"kb_used_data":5608,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752928,"statfs":{"total":773094113280,"available":772866998272,"internally_reserved":0,"allocated":5742592,"data_stored":3179920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.354926"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828699+0000","last_change":"2024-04-25T01:23:27.828699+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828699+0000","last_undegraded":"2024-04-25T01:23:27.828699+0000","last_fullsized":"2024-04-25T01:23:27.828699+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833069+0000","last_change":"2024-04-25T01:23:27.833069+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833069+0000","last_undegraded":"2024-04-25T01:23:27.833069+0000","last_fullsized":"2024-04-25T01:23:27.833069+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838229+0000","last_change":"2024-04-25T01:23:27.838229+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838229+0000","last_undegraded":"2024-04-25T01:23:27.838229+0000","last_fullsized":"2024-04-25T01:23:27.838229+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830113+0000","last_change":"2024-04-25T01:23:27.830113+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830113+0000","last_undegraded":"2024-04-25T01:23:27.830113+0000","last_fullsized":"2024-04-25T01:23:27.830113+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830687+0000","last_change":"2024-04-25T01:23:27.830687+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830687+0000","last_undegraded":"2024-04-25T01:23:27.830687+0000","last_fullsized":"2024-04-25T01:23:27.830687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828146+0000","last_change":"2024-04-25T01:23:27.828146+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828146+0000","last_undegraded":"2024-04-25T01:23:27.828146+0000","last_fullsized":"2024-04-25T01:23:27.828146+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838894+0000","last_change":"2024-04-25T01:23:27.838894+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838894+0000","last_undegraded":"2024-04-25T01:23:27.838894+0000","last_fullsized":"2024-04-25T01:23:27.838894+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834175+0000","last_change":"2024-04-25T01:23:27.834175+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834175+0000","last_undegraded":"2024-04-25T01:23:27.834175+0000","last_fullsized":"2024-04-25T01:23:27.834175+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836796+0000","last_change":"2024-04-25T01:23:27.836796+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836796+0000","last_undegraded":"2024-04-25T01:23:27.836796+0000","last_fullsized":"2024-04-25T01:23:27.836796+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827603+0000","last_change":"2024-04-25T01:23:27.827603+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827603+0000","last_undegraded":"2024-04-25T01:23:27.827603+0000","last_fullsized":"2024-04-25T01:23:27.827603+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830254+0000","last_change":"2024-04-25T01:23:27.830254+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830254+0000","last_undegraded":"2024-04-25T01:23:27.830254+0000","last_fullsized":"2024-04-25T01:23:27.830254+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833652+0000","last_change":"2024-04-25T01:23:27.833652+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833652+0000","last_undegraded":"2024-04-25T01:23:27.833652+0000","last_fullsized":"2024-04-25T01:23:27.833652+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827511+0000","last_change":"2024-04-25T01:23:27.827511+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827511+0000","last_undegraded":"2024-04-25T01:23:27.827511+0000","last_fullsized":"2024-04-25T01:23:27.827511+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838536+0000","last_change":"2024-04-25T01:23:27.838536+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838536+0000","last_undegraded":"2024-04-25T01:23:27.838536+0000","last_fullsized":"2024-04-25T01:23:27.838536+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.829534+0000","last_change":"2024-04-25T01:23:27.829534+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.829534+0000","last_undegraded":"2024-04-25T01:23:27.829534+0000","last_fullsized":"2024-04-25T01:23:27.829534+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.825966+0000","last_change":"2024-04-25T01:23:27.825966+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.825966+0000","last_undegraded":"2024-04-25T01:23:27.825966+0000","last_fullsized":"2024-04-25T01:23:27.825966+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838142+0000","last_change":"2024-04-25T01:23:27.838142+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838142+0000","last_undegraded":"2024-04-25T01:23:27.838142+0000","last_fullsized":"2024-04-25T01:23:27.838142+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832007+0000","last_change":"2024-04-25T01:23:27.832007+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832007+0000","last_undegraded":"2024-04-25T01:23:27.832007+0000","last_fullsized":"2024-04-25T01:23:27.832007+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836983+0000","last_change":"2024-04-25T01:23:27.836983+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836983+0000","last_undegraded":"2024-04-25T01:23:27.836983+0000","last_fullsized":"2024-04-25T01:23:27.836983+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832039+0000","last_change":"2024-04-25T01:23:27.832039+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832039+0000","last_undegraded":"2024-04-25T01:23:27.832039+0000","last_fullsized":"2024-04-25T01:23:27.832039+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830258+0000","last_change":"2024-04-25T01:23:27.830258+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830258+0000","last_undegraded":"2024-04-25T01:23:27.830258+0000","last_fullsized":"2024-04-25T01:23:27.830258+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833623+0000","last_change":"2024-04-25T01:23:27.833623+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833623+0000","last_undegraded":"2024-04-25T01:23:27.833623+0000","last_fullsized":"2024-04-25T01:23:27.833623+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830382+0000","last_change":"2024-04-25T01:23:27.830382+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830382+0000","last_undegraded":"2024-04-25T01:23:27.830382+0000","last_fullsized":"2024-04-25T01:23:27.830382+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836595+0000","last_change":"2024-04-25T01:23:27.836595+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836595+0000","last_undegraded":"2024-04-25T01:23:27.836595+0000","last_fullsized":"2024-04-25T01:23:27.836595+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836402+0000","last_change":"2024-04-25T01:23:27.836402+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836402+0000","last_undegraded":"2024-04-25T01:23:27.836402+0000","last_fullsized":"2024-04-25T01:23:27.836402+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835991+0000","last_change":"2024-04-25T01:23:27.835991+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835991+0000","last_undegraded":"2024-04-25T01:23:27.835991+0000","last_fullsized":"2024-04-25T01:23:27.835991+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827570+0000","last_change":"2024-04-25T01:23:27.827570+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827570+0000","last_undegraded":"2024-04-25T01:23:27.827570+0000","last_fullsized":"2024-04-25T01:23:27.827570+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831597+0000","last_change":"2024-04-25T01:23:27.831597+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831597+0000","last_undegraded":"2024-04-25T01:23:27.831597+0000","last_fullsized":"2024-04-25T01:23:27.831597+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836573+0000","last_change":"2024-04-25T01:23:27.836573+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836573+0000","last_undegraded":"2024-04-25T01:23:27.836573+0000","last_fullsized":"2024-04-25T01:23:27.836573+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830474+0000","last_change":"2024-04-25T01:23:27.830474+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830474+0000","last_undegraded":"2024-04-25T01:23:27.830474+0000","last_fullsized":"2024-04-25T01:23:27.830474+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835732+0000","last_change":"2024-04-25T01:23:27.835732+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835732+0000","last_undegraded":"2024-04-25T01:23:27.835732+0000","last_fullsized":"2024-04-25T01:23:27.835732+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828172+0000","last_change":"2024-04-25T01:23:27.828172+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828172+0000","last_undegraded":"2024-04-25T01:23:27.828172+0000","last_fullsized":"2024-04-25T01:23:27.828172+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830914+0000","last_change":"2024-04-25T01:23:27.830914+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830914+0000","last_undegraded":"2024-04-25T01:23:27.830914+0000","last_fullsized":"2024-04-25T01:23:27.830914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844780+0000","last_change":"2024-04-25T01:23:27.840369+0000","last_active":"2024-04-25T01:23:27.844780+0000","last_peered":"2024-04-25T01:23:27.844780+0000","last_clean":"2024-04-25T01:23:27.844780+0000","last_became_active":"2024-04-25T01:23:27.839987+0000","last_became_peered":"2024-04-25T01:23:27.839987+0000","last_unstale":"2024-04-25T01:23:27.844780+0000","last_undegraded":"2024-04-25T01:23:27.844780+0000","last_fullsized":"2024-04-25T01:23:27.844780+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.219905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826648+0000","last_change":"2024-04-25T01:23:27.826648+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826648+0000","last_undegraded":"2024-04-25T01:23:27.826648+0000","last_fullsized":"2024-04-25T01:23:27.826648+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.841990+0000","last_change":"2024-04-25T01:23:27.838394+0000","last_active":"2024-04-25T01:23:27.841990+0000","last_peered":"2024-04-25T01:23:27.841990+0000","last_clean":"2024-04-25T01:23:27.841990+0000","last_became_active":"2024-04-25T01:23:27.838114+0000","last_became_peered":"2024-04-25T01:23:27.838114+0000","last_unstale":"2024-04-25T01:23:27.841990+0000","last_undegraded":"2024-04-25T01:23:27.841990+0000","last_fullsized":"2024-04-25T01:23:27.841990+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:48.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832822+0000","last_change":"2024-04-25T01:23:27.832822+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832822+0000","last_undegraded":"2024-04-25T01:23:27.832822+0000","last_fullsized":"2024-04-25T01:23:27.832822+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.842100+0000","last_change":"2024-04-25T01:23:27.839860+0000","last_active":"2024-04-25T01:23:27.842100+0000","last_peered":"2024-04-25T01:23:27.842100+0000","last_clean":"2024-04-25T01:23:27.842100+0000","last_became_active":"2024-04-25T01:23:27.839589+0000","last_became_peered":"2024-04-25T01:23:27.839589+0000","last_unstale":"2024-04-25T01:23:27.842100+0000","last_undegraded":"2024-04-25T01:23:27.842100+0000","last_fullsized":"2024-04-25T01:23:27.842100+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:52.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835561+0000","last_change":"2024-04-25T01:23:27.835561+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835561+0000","last_undegraded":"2024-04-25T01:23:27.835561+0000","last_fullsized":"2024-04-25T01:23:27.835561+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847649+0000","last_change":"2024-04-25T01:23:27.842142+0000","last_active":"2024-04-25T01:23:27.847649+0000","last_peered":"2024-04-25T01:23:27.847649+0000","last_clean":"2024-04-25T01:23:27.847649+0000","last_became_active":"2024-04-25T01:23:27.841740+0000","last_became_peered":"2024-04-25T01:23:27.841740+0000","last_unstale":"2024-04-25T01:23:27.847649+0000","last_undegraded":"2024-04-25T01:23:27.847649+0000","last_fullsized":"2024-04-25T01:23:27.847649+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:13.292186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831025+0000","last_change":"2024-04-25T01:23:27.831025+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831025+0000","last_undegraded":"2024-04-25T01:23:27.831025+0000","last_fullsized":"2024-04-25T01:23:27.831025+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844533+0000","last_change":"2024-04-25T01:23:27.831214+0000","last_active":"2024-04-25T01:23:27.844533+0000","last_peered":"2024-04-25T01:23:27.844533+0000","last_clean":"2024-04-25T01:23:27.844533+0000","last_became_active":"2024-04-25T01:23:27.830883+0000","last_became_peered":"2024-04-25T01:23:27.830883+0000","last_unstale":"2024-04-25T01:23:27.844533+0000","last_undegraded":"2024-04-25T01:23:27.844533+0000","last_fullsized":"2024-04-25T01:23:27.844533+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:40.061850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830940+0000","last_change":"2024-04-25T01:23:27.830940+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830940+0000","last_undegraded":"2024-04-25T01:23:27.830940+0000","last_fullsized":"2024-04-25T01:23:27.830940+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844655+0000","last_change":"2024-04-25T01:23:27.839506+0000","last_active":"2024-04-25T01:23:27.844655+0000","last_peered":"2024-04-25T01:23:27.844655+0000","last_clean":"2024-04-25T01:23:27.844655+0000","last_became_active":"2024-04-25T01:23:27.838600+0000","last_became_peered":"2024-04-25T01:23:27.838600+0000","last_unstale":"2024-04-25T01:23:27.844655+0000","last_undegraded":"2024-04-25T01:23:27.844655+0000","last_fullsized":"2024-04-25T01:23:27.844655+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:48.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833846+0000","last_change":"2024-04-25T01:23:27.833846+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833846+0000","last_undegraded":"2024-04-25T01:23:27.833846+0000","last_fullsized":"2024-04-25T01:23:27.833846+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847468+0000","last_change":"2024-04-25T01:23:27.843276+0000","last_active":"2024-04-25T01:23:27.847468+0000","last_peered":"2024-04-25T01:23:27.847468+0000","last_clean":"2024-04-25T01:23:27.847468+0000","last_became_active":"2024-04-25T01:23:27.842824+0000","last_became_peered":"2024-04-25T01:23:27.842824+0000","last_unstale":"2024-04-25T01:23:27.847468+0000","last_undegraded":"2024-04-25T01:23:27.847468+0000","last_fullsized":"2024-04-25T01:23:27.847468+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:48.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831669+0000","last_change":"2024-04-25T01:23:27.831669+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831669+0000","last_undegraded":"2024-04-25T01:23:27.831669+0000","last_fullsized":"2024-04-25T01:23:27.831669+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848229+0000","last_change":"2024-04-25T01:23:27.842530+0000","last_active":"2024-04-25T01:23:27.848229+0000","last_peered":"2024-04-25T01:23:27.848229+0000","last_clean":"2024-04-25T01:23:27.848229+0000","last_became_active":"2024-04-25T01:23:27.842265+0000","last_became_peered":"2024-04-25T01:23:27.842265+0000","last_unstale":"2024-04-25T01:23:27.848229+0000","last_undegraded":"2024-04-25T01:23:27.848229+0000","last_fullsized":"2024-04-25T01:23:27.848229+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:49:28.578991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.829914+0000","last_change":"2024-04-25T01:23:27.829914+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.829914+0000","last_undegraded":"2024-04-25T01:23:27.829914+0000","last_fullsized":"2024-04-25T01:23:27.829914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847439+0000","last_change":"2024-04-25T01:23:27.842714+0000","last_active":"2024-04-25T01:23:27.847439+0000","last_peered":"2024-04-25T01:23:27.847439+0000","last_clean":"2024-04-25T01:23:27.847439+0000","last_became_active":"2024-04-25T01:23:27.842268+0000","last_became_peered":"2024-04-25T01:23:27.842268+0000","last_unstale":"2024-04-25T01:23:27.847439+0000","last_undegraded":"2024-04-25T01:23:27.847439+0000","last_fullsized":"2024-04-25T01:23:27.847439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:01.477667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827321+0000","last_change":"2024-04-25T01:23:27.827321+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827321+0000","last_undegraded":"2024-04-25T01:23:27.827321+0000","last_fullsized":"2024-04-25T01:23:27.827321+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844619+0000","last_change":"2024-04-25T01:23:27.839254+0000","last_active":"2024-04-25T01:23:27.844619+0000","last_peered":"2024-04-25T01:23:27.844619+0000","last_clean":"2024-04-25T01:23:27.844619+0000","last_became_active":"2024-04-25T01:23:27.838972+0000","last_became_peered":"2024-04-25T01:23:27.838972+0000","last_unstale":"2024-04-25T01:23:27.844619+0000","last_undegraded":"2024-04-25T01:23:27.844619+0000","last_fullsized":"2024-04-25T01:23:27.844619+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:04.719084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834497+0000","last_change":"2024-04-25T01:23:27.834497+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834497+0000","last_undegraded":"2024-04-25T01:23:27.834497+0000","last_fullsized":"2024-04-25T01:23:27.834497+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847498+0000","last_change":"2024-04-25T01:23:27.843358+0000","last_active":"2024-04-25T01:23:27.847498+0000","last_peered":"2024-04-25T01:23:27.847498+0000","last_clean":"2024-04-25T01:23:27.847498+0000","last_became_active":"2024-04-25T01:23:27.843098+0000","last_became_peered":"2024-04-25T01:23:27.843098+0000","last_unstale":"2024-04-25T01:23:27.847498+0000","last_undegraded":"2024-04-25T01:23:27.847498+0000","last_fullsized":"2024-04-25T01:23:27.847498+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:38.211801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830273+0000","last_change":"2024-04-25T01:23:27.830273+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830273+0000","last_undegraded":"2024-04-25T01:23:27.830273+0000","last_fullsized":"2024-04-25T01:23:27.830273+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846301+0000","last_change":"2024-04-25T01:23:27.841668+0000","last_active":"2024-04-25T01:23:27.846301+0000","last_peered":"2024-04-25T01:23:27.846301+0000","last_clean":"2024-04-25T01:23:27.846301+0000","last_became_active":"2024-04-25T01:23:27.841382+0000","last_became_peered":"2024-04-25T01:23:27.841382+0000","last_unstale":"2024-04-25T01:23:27.846301+0000","last_undegraded":"2024-04-25T01:23:27.846301+0000","last_fullsized":"2024-04-25T01:23:27.846301+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:00:11.736192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827805+0000","last_change":"2024-04-25T01:23:27.827805+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827805+0000","last_undegraded":"2024-04-25T01:23:27.827805+0000","last_fullsized":"2024-04-25T01:23:27.827805+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848483+0000","last_change":"2024-04-25T01:23:27.845237+0000","last_active":"2024-04-25T01:23:27.848483+0000","last_peered":"2024-04-25T01:23:27.848483+0000","last_clean":"2024-04-25T01:23:27.848483+0000","last_became_active":"2024-04-25T01:23:27.844336+0000","last_became_peered":"2024-04-25T01:23:27.844336+0000","last_unstale":"2024-04-25T01:23:27.848483+0000","last_undegraded":"2024-04-25T01:23:27.848483+0000","last_fullsized":"2024-04-25T01:23:27.848483+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:52.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834532+0000","last_change":"2024-04-25T01:23:27.834532+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834532+0000","last_undegraded":"2024-04-25T01:23:27.834532+0000","last_fullsized":"2024-04-25T01:23:27.834532+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.849509+0000","last_change":"2024-04-25T01:23:27.848962+0000","last_active":"2024-04-25T01:23:27.849509+0000","last_peered":"2024-04-25T01:23:27.849509+0000","last_clean":"2024-04-25T01:23:27.849509+0000","last_became_active":"2024-04-25T01:23:27.848694+0000","last_became_peered":"2024-04-25T01:23:27.848694+0000","last_unstale":"2024-04-25T01:23:27.849509+0000","last_undegraded":"2024-04-25T01:23:27.849509+0000","last_fullsized":"2024-04-25T01:23:27.849509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:04.282731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834242+0000","last_change":"2024-04-25T01:23:27.834242+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834242+0000","last_undegraded":"2024-04-25T01:23:27.834242+0000","last_fullsized":"2024-04-25T01:23:27.834242+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847532+0000","last_change":"2024-04-25T01:23:27.843254+0000","last_active":"2024-04-25T01:23:27.847532+0000","last_peered":"2024-04-25T01:23:27.847532+0000","last_clean":"2024-04-25T01:23:27.847532+0000","last_became_active":"2024-04-25T01:23:27.842506+0000","last_became_peered":"2024-04-25T01:23:27.842506+0000","last_unstale":"2024-04-25T01:23:27.847532+0000","last_undegraded":"2024-04-25T01:23:27.847532+0000","last_fullsized":"2024-04-25T01:23:27.847532+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:42.248852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835026+0000","last_change":"2024-04-25T01:23:27.835026+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835026+0000","last_undegraded":"2024-04-25T01:23:27.835026+0000","last_fullsized":"2024-04-25T01:23:27.835026+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846277+0000","last_change":"2024-04-25T01:23:27.841058+0000","last_active":"2024-04-25T01:23:27.846277+0000","last_peered":"2024-04-25T01:23:27.846277+0000","last_clean":"2024-04-25T01:23:27.846277+0000","last_became_active":"2024-04-25T01:23:27.840808+0000","last_became_peered":"2024-04-25T01:23:27.840808+0000","last_unstale":"2024-04-25T01:23:27.846277+0000","last_undegraded":"2024-04-25T01:23:27.846277+0000","last_fullsized":"2024-04-25T01:23:27.846277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:03.437919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833035+0000","last_change":"2024-04-25T01:23:27.833035+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833035+0000","last_undegraded":"2024-04-25T01:23:27.833035+0000","last_fullsized":"2024-04-25T01:23:27.833035+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848509+0000","last_change":"2024-04-25T01:23:27.846019+0000","last_active":"2024-04-25T01:23:27.848509+0000","last_peered":"2024-04-25T01:23:27.848509+0000","last_clean":"2024-04-25T01:23:27.848509+0000","last_became_active":"2024-04-25T01:23:27.845056+0000","last_became_peered":"2024-04-25T01:23:27.845056+0000","last_unstale":"2024-04-25T01:23:27.848509+0000","last_undegraded":"2024-04-25T01:23:27.848509+0000","last_fullsized":"2024-04-25T01:23:27.848509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:08.739387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826903+0000","last_change":"2024-04-25T01:23:27.826903+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826903+0000","last_undegraded":"2024-04-25T01:23:27.826903+0000","last_fullsized":"2024-04-25T01:23:27.826903+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.845756+0000","last_change":"2024-04-25T01:23:27.839989+0000","last_active":"2024-04-25T01:23:27.845756+0000","last_peered":"2024-04-25T01:23:27.845756+0000","last_clean":"2024-04-25T01:23:27.845756+0000","last_became_active":"2024-04-25T01:23:27.839737+0000","last_became_peered":"2024-04-25T01:23:27.839737+0000","last_unstale":"2024-04-25T01:23:27.845756+0000","last_undegraded":"2024-04-25T01:23:27.845756+0000","last_fullsized":"2024-04-25T01:23:27.845756+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:17.100820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830591+0000","last_change":"2024-04-25T01:23:27.830591+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830591+0000","last_undegraded":"2024-04-25T01:23:27.830591+0000","last_fullsized":"2024-04-25T01:23:27.830591+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.845062+0000","last_change":"2024-04-25T01:23:27.838563+0000","last_active":"2024-04-25T01:23:27.845062+0000","last_peered":"2024-04-25T01:23:27.845062+0000","last_clean":"2024-04-25T01:23:27.845062+0000","last_became_active":"2024-04-25T01:23:27.837764+0000","last_became_peered":"2024-04-25T01:23:27.837764+0000","last_unstale":"2024-04-25T01:23:27.845062+0000","last_undegraded":"2024-04-25T01:23:27.845062+0000","last_fullsized":"2024-04-25T01:23:27.845062+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:26.874008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826397+0000","last_change":"2024-04-25T01:23:27.826397+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826397+0000","last_undegraded":"2024-04-25T01:23:27.826397+0000","last_fullsized":"2024-04-25T01:23:27.826397+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847457+0000","last_change":"2024-04-25T01:23:27.843101+0000","last_active":"2024-04-25T01:23:27.847457+0000","last_peered":"2024-04-25T01:23:27.847457+0000","last_clean":"2024-04-25T01:23:27.847457+0000","last_became_active":"2024-04-25T01:23:27.842810+0000","last_became_peered":"2024-04-25T01:23:27.842810+0000","last_unstale":"2024-04-25T01:23:27.847457+0000","last_undegraded":"2024-04-25T01:23:27.847457+0000","last_fullsized":"2024-04-25T01:23:27.847457+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:04.719084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826783+0000","last_change":"2024-04-25T01:23:27.826783+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826783+0000","last_undegraded":"2024-04-25T01:23:27.826783+0000","last_fullsized":"2024-04-25T01:23:27.826783+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.838395+0000","last_change":"2024-04-25T01:23:27.834285+0000","last_active":"2024-04-25T01:23:27.838395+0000","last_peered":"2024-04-25T01:23:27.838395+0000","last_clean":"2024-04-25T01:23:27.838395+0000","last_became_active":"2024-04-25T01:23:27.834030+0000","last_became_peered":"2024-04-25T01:23:27.834030+0000","last_unstale":"2024-04-25T01:23:27.838395+0000","last_undegraded":"2024-04-25T01:23:27.838395+0000","last_fullsized":"2024-04-25T01:23:27.838395+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:33.027603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832642+0000","last_change":"2024-04-25T01:23:27.832642+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832642+0000","last_undegraded":"2024-04-25T01:23:27.832642+0000","last_fullsized":"2024-04-25T01:23:27.832642+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846225+0000","last_change":"2024-04-25T01:23:27.839744+0000","last_active":"2024-04-25T01:23:27.846225+0000","last_peered":"2024-04-25T01:23:27.846225+0000","last_clean":"2024-04-25T01:23:27.846225+0000","last_became_active":"2024-04-25T01:23:27.839478+0000","last_became_peered":"2024-04-25T01:23:27.839478+0000","last_unstale":"2024-04-25T01:23:27.846225+0000","last_undegraded":"2024-04-25T01:23:27.846225+0000","last_fullsized":"2024-04-25T01:23:27.846225+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:44.264978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833162+0000","last_change":"2024-04-25T01:23:27.833162+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833162+0000","last_undegraded":"2024-04-25T01:23:27.833162+0000","last_fullsized":"2024-04-25T01:23:27.833162+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848293+0000","last_change":"2024-04-25T01:23:27.843276+0000","last_active":"2024-04-25T01:23:27.848293+0000","last_peered":"2024-04-25T01:23:27.848293+0000","last_clean":"2024-04-25T01:23:27.848293+0000","last_became_active":"2024-04-25T01:23:27.842913+0000","last_became_peered":"2024-04-25T01:23:27.842913+0000","last_unstale":"2024-04-25T01:23:27.848293+0000","last_undegraded":"2024-04-25T01:23:27.848293+0000","last_fullsized":"2024-04-25T01:23:27.848293+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:33.027603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826481+0000","last_change":"2024-04-25T01:23:27.826481+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826481+0000","last_undegraded":"2024-04-25T01:23:27.826481+0000","last_fullsized":"2024-04-25T01:23:27.826481+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846474+0000","last_change":"2024-04-25T01:23:27.843723+0000","last_active":"2024-04-25T01:23:27.846474+0000","last_peered":"2024-04-25T01:23:27.846474+0000","last_clean":"2024-04-25T01:23:27.846474+0000","last_became_active":"2024-04-25T01:23:27.843464+0000","last_became_peered":"2024-04-25T01:23:27.843464+0000","last_unstale":"2024-04-25T01:23:27.846474+0000","last_undegraded":"2024-04-25T01:23:27.846474+0000","last_fullsized":"2024-04-25T01:23:27.846474+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:01.926860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826655+0000","last_change":"2024-04-25T01:23:27.826655+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826655+0000","last_undegraded":"2024-04-25T01:23:27.826655+0000","last_fullsized":"2024-04-25T01:23:27.826655+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848098+0000","last_change":"2024-04-25T01:23:27.847328+0000","last_active":"2024-04-25T01:23:27.848098+0000","last_peered":"2024-04-25T01:23:27.848098+0000","last_clean":"2024-04-25T01:23:27.848098+0000","last_became_active":"2024-04-25T01:23:27.847046+0000","last_became_peered":"2024-04-25T01:23:27.847046+0000","last_unstale":"2024-04-25T01:23:27.848098+0000","last_undegraded":"2024-04-25T01:23:27.848098+0000","last_fullsized":"2024-04-25T01:23:27.848098+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:59.145724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827287+0000","last_change":"2024-04-25T01:23:27.827287+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827287+0000","last_undegraded":"2024-04-25T01:23:27.827287+0000","last_fullsized":"2024-04-25T01:23:27.827287+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847476+0000","last_change":"2024-04-25T01:23:27.842809+0000","last_active":"2024-04-25T01:23:27.847476+0000","last_peered":"2024-04-25T01:23:27.847476+0000","last_clean":"2024-04-25T01:23:27.847476+0000","last_became_active":"2024-04-25T01:23:27.842561+0000","last_became_peered":"2024-04-25T01:23:27.842561+0000","last_unstale":"2024-04-25T01:23:27.847476+0000","last_undegraded":"2024-04-25T01:23:27.847476+0000","last_fullsized":"2024-04-25T01:23:27.847476+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:28:34.927620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835126+0000","last_change":"2024-04-25T01:23:27.835126+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835126+0000","last_undegraded":"2024-04-25T01:23:27.835126+0000","last_fullsized":"2024-04-25T01:23:27.835126+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848367+0000","last_change":"2024-04-25T01:23:27.844808+0000","last_active":"2024-04-25T01:23:27.848367+0000","last_peered":"2024-04-25T01:23:27.848367+0000","last_clean":"2024-04-25T01:23:27.848367+0000","last_became_active":"2024-04-25T01:23:27.844541+0000","last_became_peered":"2024-04-25T01:23:27.844541+0000","last_unstale":"2024-04-25T01:23:27.848367+0000","last_undegraded":"2024-04-25T01:23:27.848367+0000","last_fullsized":"2024-04-25T01:23:27.848367+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:48.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832949+0000","last_change":"2024-04-25T01:23:27.832949+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832949+0000","last_undegraded":"2024-04-25T01:23:27.832949+0000","last_fullsized":"2024-04-25T01:23:27.832949+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846394+0000","last_change":"2024-04-25T01:23:27.842454+0000","last_active":"2024-04-25T01:23:27.846394+0000","last_peered":"2024-04-25T01:23:27.846394+0000","last_clean":"2024-04-25T01:23:27.846394+0000","last_became_active":"2024-04-25T01:23:27.842209+0000","last_became_peered":"2024-04-25T01:23:27.842209+0000","last_unstale":"2024-04-25T01:23:27.846394+0000","last_undegraded":"2024-04-25T01:23:27.846394+0000","last_fullsized":"2024-04-25T01:23:27.846394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:22.705548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826687+0000","last_change":"2024-04-25T01:23:27.826687+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826687+0000","last_undegraded":"2024-04-25T01:23:27.826687+0000","last_fullsized":"2024-04-25T01:23:27.826687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846248+0000","last_change":"2024-04-25T01:23:27.841080+0000","last_active":"2024-04-25T01:23:27.846248+0000","last_peered":"2024-04-25T01:23:27.846248+0000","last_clean":"2024-04-25T01:23:27.846248+0000","last_became_active":"2024-04-25T01:23:27.840813+0000","last_became_peered":"2024-04-25T01:23:27.840813+0000","last_unstale":"2024-04-25T01:23:27.846248+0000","last_undegraded":"2024-04-25T01:23:27.846248+0000","last_fullsized":"2024-04-25T01:23:27.846248+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:44.720263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833838+0000","last_change":"2024-04-25T01:23:27.833838+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833838+0000","last_undegraded":"2024-04-25T01:23:27.833838+0000","last_fullsized":"2024-04-25T01:23:27.833838+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844601+0000","last_change":"2024-04-25T01:23:27.841167+0000","last_active":"2024-04-25T01:23:27.844601+0000","last_peered":"2024-04-25T01:23:27.844601+0000","last_clean":"2024-04-25T01:23:27.844601+0000","last_became_active":"2024-04-25T01:23:27.840932+0000","last_became_peered":"2024-04-25T01:23:27.840932+0000","last_unstale":"2024-04-25T01:23:27.844601+0000","last_undegraded":"2024-04-25T01:23:27.844601+0000","last_fullsized":"2024-04-25T01:23:27.844601+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:59.145724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836435+0000","last_change":"2024-04-25T01:23:27.836435+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836435+0000","last_undegraded":"2024-04-25T01:23:27.836435+0000","last_fullsized":"2024-04-25T01:23:27.836435+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.842097+0000","last_change":"2024-04-25T01:23:27.838507+0000","last_active":"2024-04-25T01:23:27.842097+0000","last_peered":"2024-04-25T01:23:27.842097+0000","last_clean":"2024-04-25T01:23:27.842097+0000","last_became_active":"2024-04-25T01:23:27.838016+0000","last_became_peered":"2024-04-25T01:23:27.838016+0000","last_unstale":"2024-04-25T01:23:27.842097+0000","last_undegraded":"2024-04-25T01:23:27.842097+0000","last_fullsized":"2024-04-25T01:23:27.842097+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:09.155513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831341+0000","last_change":"2024-04-25T01:23:27.831341+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831341+0000","last_undegraded":"2024-04-25T01:23:27.831341+0000","last_fullsized":"2024-04-25T01:23:27.831341+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846638+0000","last_change":"2024-04-25T01:23:27.841576+0000","last_active":"2024-04-25T01:23:27.846638+0000","last_peered":"2024-04-25T01:23:27.846638+0000","last_clean":"2024-04-25T01:23:27.846638+0000","last_became_active":"2024-04-25T01:23:27.839038+0000","last_became_peered":"2024-04-25T01:23:27.839038+0000","last_unstale":"2024-04-25T01:23:27.846638+0000","last_undegraded":"2024-04-25T01:23:27.846638+0000","last_fullsized":"2024-04-25T01:23:27.846638+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:26:54.800425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":222,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.833349+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:23:27.833349+0000","last_peered":"2024-04-25T01:23:27.833349+0000","last_clean":"2024-04-25T01:23:27.833349+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:23:27.833349+0000","last_undegraded":"2024-04-25T01:23:27.833349+0000","last_fullsized":"2024-04-25T01:23:27.833349+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1163264,"data_stored":840266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1163264,"data_stored":840266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:23:32.127 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:23:32.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.279+0000 7f7c2e709640 1 Processor -- start 2024-04-25T01:23:32.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.279+0000 7f7c2e709640 1 -- start start 2024-04-25T01:23:32.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.279+0000 7f7c2e709640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c2815e4c0 0x7f7c280d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.279+0000 7f7c2e709640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c280d2d40 0x7f7c280cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.279+0000 7f7c2e709640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c280d1e90 con 0x7f7c280d4a10 2024-04-25T01:23:32.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.279+0000 7f7c2e709640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7c280b54a0 con 0x7f7c280d2d40 2024-04-25T01:23:32.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.279+0000 7f7c2e709640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7c280d2130 con 0x7f7c2815e4c0 2024-04-25T01:23:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.280+0000 7f7c27fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c2815e4c0 0x7f7c280d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.280+0000 7f7c2cca6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c280d2d40 0x7f7c280cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.280+0000 7f7c27fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c2815e4c0 0x7f7c280d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:37090/0 (socket says 172.21.15.77:37090) 2024-04-25T01:23:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.280+0000 7f7c27fff640 1 -- 172.21.15.77:0/2648412210 learned_addr learned my addr 172.21.15.77:0/2648412210 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.280+0000 7f7c2cca6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c280d2d40 0x7f7c280cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:41048/0 (socket says 172.21.15.77:41048) 2024-04-25T01:23:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.280+0000 7f7c277fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7c280d4a10 0x7f7c280b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:49146/0 (socket says 172.21.15.77:49146) 2024-04-25T01:23:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.280+0000 7f7c27fff640 1 -- 172.21.15.77:0/2648412210 >> v2:172.21.15.121:3300/0 conn(0x7f7c280d2d40 msgr2=0x7f7c280cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.280+0000 7f7c27fff640 1 --2- 172.21.15.77:0/2648412210 >> v2:172.21.15.121:3300/0 conn(0x7f7c280d2d40 0x7f7c280cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.280+0000 7f7c27fff640 1 -- 172.21.15.77:0/2648412210 >> v1:172.21.15.121:6789/0 conn(0x7f7c280d4a10 legacy=0x7f7c280b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.280+0000 7f7c27fff640 1 -- 172.21.15.77:0/2648412210 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c280b5780 con 0x7f7c2815e4c0 2024-04-25T01:23:32.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.280+0000 7f7c2cca6640 1 --2- 172.21.15.77:0/2648412210 >> v2:172.21.15.121:3300/0 conn(0x7f7c280d2d40 0x7f7c280cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:32.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.281+0000 7f7c27fff640 1 --2- 172.21.15.77:0/2648412210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c2815e4c0 0x7f7c280d2800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009870 tx=0x7f7c1802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af281dd228ddd58f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.281+0000 7f7c26ffd640 1 -- 172.21.15.77:0/2648412210 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18037070 con 0x7f7c2815e4c0 2024-04-25T01:23:32.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.281+0000 7f7c26ffd640 1 -- 172.21.15.77:0/2648412210 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c18032440 con 0x7f7c2815e4c0 2024-04-25T01:23:32.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.281+0000 7f7c26ffd640 1 -- 172.21.15.77:0/2648412210 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18032810 con 0x7f7c2815e4c0 2024-04-25T01:23:32.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.281+0000 7f7c2e709640 1 -- 172.21.15.77:0/2648412210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c2815e4c0 msgr2=0x7f7c280d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.281+0000 7f7c2e709640 1 --2- 172.21.15.77:0/2648412210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c2815e4c0 0x7f7c280d2800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009870 tx=0x7f7c1802a1c0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.282+0000 7f7c2e709640 1 -- 172.21.15.77:0/2648412210 shutdown_connections 2024-04-25T01:23:32.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.282+0000 7f7c2e709640 1 --2- 172.21.15.77:0/2648412210 >> v2:172.21.15.121:3300/0 conn(0x7f7c280d2d40 0x7f7c280cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.282+0000 7f7c2e709640 1 --2- 172.21.15.77:0/2648412210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c2815e4c0 0x7f7c280d2800 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.282+0000 7f7c2e709640 1 -- 172.21.15.77:0/2648412210 >> 172.21.15.77:0/2648412210 conn(0x7f7c280d3ef0 msgr2=0x7f7c280d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:32.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.282+0000 7f7c2e709640 1 -- 172.21.15.77:0/2648412210 shutdown_connections 2024-04-25T01:23:32.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.282+0000 7f7c2e709640 1 -- 172.21.15.77:0/2648412210 wait complete. 2024-04-25T01:23:32.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.283+0000 7f7c2e709640 1 Processor -- start 2024-04-25T01:23:32.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.283+0000 7f7c2e709640 1 -- start start 2024-04-25T01:23:32.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2e709640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c280d2d40 0x7f7c2815b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2e709640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c280d4a10 0x7f7c2815bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2e709640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c2815e4c0 0x7f7c2815c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2e709640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2815ea20 con 0x7f7c2815e4c0 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2e709640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2815e8a0 con 0x7f7c280d4a10 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2e709640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7c280b45f0 con 0x7f7c280d2d40 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2cca6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c2815e4c0 0x7f7c2815c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2cca6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c2815e4c0 0x7f7c2815c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42402/0 (socket says 172.21.15.77:42402) 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2cca6640 1 -- 172.21.15.77:0/1473725223 learned_addr learned my addr 172.21.15.77:0/1473725223 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c27fff640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c280d2d40 0x7f7c2815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c277fe640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c280d4a10 0x7f7c2815bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2cca6640 1 -- 172.21.15.77:0/1473725223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c280d2d40 msgr2=0x7f7c2815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2cca6640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c280d2d40 0x7f7c2815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2cca6640 1 -- 172.21.15.77:0/1473725223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c280d4a10 msgr2=0x7f7c2815bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2cca6640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c280d4a10 0x7f7c2815bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2cca6640 1 -- 172.21.15.77:0/1473725223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c280cbc80 con 0x7f7c2815e4c0 2024-04-25T01:23:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c2cca6640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c2815e4c0 0x7f7c2815c180 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000efc0 tx=0x7f7c20009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.284+0000 7f7c24ff9640 1 -- 172.21.15.77:0/1473725223 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c20018030 con 0x7f7c2815e4c0 2024-04-25T01:23:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.285+0000 7f7c24ff9640 1 -- 172.21.15.77:0/1473725223 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c2000cde0 con 0x7f7c2815e4c0 2024-04-25T01:23:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.285+0000 7f7c2e709640 1 -- 172.21.15.77:0/1473725223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c280cbf60 con 0x7f7c2815e4c0 2024-04-25T01:23:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.285+0000 7f7c24ff9640 1 -- 172.21.15.77:0/1473725223 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c20003bb0 con 0x7f7c2815e4c0 2024-04-25T01:23:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.285+0000 7f7c2e709640 1 -- 172.21.15.77:0/1473725223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c280cc520 con 0x7f7c2815e4c0 2024-04-25T01:23:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.286+0000 7f7c2e709640 1 -- 172.21.15.77:0/1473725223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c28068fb0 con 0x7f7c2815e4c0 2024-04-25T01:23:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.286+0000 7f7c24ff9640 1 -- 172.21.15.77:0/1473725223 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7c2001b070 con 0x7f7c2815e4c0 2024-04-25T01:23:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.286+0000 7f7c24ff9640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c1406be30 0x7f7c1406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.286+0000 7f7c24ff9640 1 -- 172.21.15.77:0/1473725223 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f7c2008b8d0 con 0x7f7c2815e4c0 2024-04-25T01:23:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.286+0000 7f7c27fff640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c1406be30 0x7f7c1406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.287+0000 7f7c27fff640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c1406be30 0x7f7c1406e2f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7c18034000 tx=0x7f7c1802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.291+0000 7f7c24ff9640 1 -- 172.21.15.77:0/1473725223 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7c20010040 con 0x7f7c2815e4c0 2024-04-25T01:23:32.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.460+0000 7f7c2e709640 1 -- 172.21.15.77:0/1473725223 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7c280b2bc0 con 0x7f7c1406be30 2024-04-25T01:23:32.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.466+0000 7f7c24ff9640 1 -- 172.21.15.77:0/1473725223 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246638 (secure 0 0 0) 0x7f7c280b2bc0 con 0x7f7c1406be30 2024-04-25T01:23:32.450 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:32.451 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:23:32.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 -- 172.21.15.77:0/1473725223 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c1406be30 msgr2=0x7f7c1406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c1406be30 0x7f7c1406e2f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7c18034000 tx=0x7f7c1802e000 comp rx=0 tx=0).stop 2024-04-25T01:23:32.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 -- 172.21.15.77:0/1473725223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c2815e4c0 msgr2=0x7f7c2815c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c2815e4c0 0x7f7c2815c180 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000efc0 tx=0x7f7c20009f90 comp rx=0 tx=0).stop 2024-04-25T01:23:32.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 -- 172.21.15.77:0/1473725223 shutdown_connections 2024-04-25T01:23:32.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7c1406be30 0x7f7c1406e2f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c2815e4c0 0x7f7c2815c180 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c280d4a10 0x7f7c2815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 --2- 172.21.15.77:0/1473725223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c280d2d40 0x7f7c2815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 -- 172.21.15.77:0/1473725223 >> 172.21.15.77:0/1473725223 conn(0x7f7c280d3ef0 msgr2=0x7f7c280d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:32.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 -- 172.21.15.77:0/1473725223 shutdown_connections 2024-04-25T01:23:32.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.469+0000 7f7c2e709640 1 -- 172.21.15.77:0/1473725223 wait complete. 2024-04-25T01:23:32.461 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-04-25T01:23:31.176800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221792,"kb_used_data":5608,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752928,"statfs":{"total":773094113280,"available":772866998272,"internally_reserved":0,"allocated":5742592,"data_stored":3179920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.354926"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828699+0000","last_change":"2024-04-25T01:23:27.828699+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828699+0000","last_undegraded":"2024-04-25T01:23:27.828699+0000","last_fullsized":"2024-04-25T01:23:27.828699+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833069+0000","last_change":"2024-04-25T01:23:27.833069+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833069+0000","last_undegraded":"2024-04-25T01:23:27.833069+0000","last_fullsized":"2024-04-25T01:23:27.833069+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838229+0000","last_change":"2024-04-25T01:23:27.838229+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838229+0000","last_undegraded":"2024-04-25T01:23:27.838229+0000","last_fullsized":"2024-04-25T01:23:27.838229+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830113+0000","last_change":"2024-04-25T01:23:27.830113+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830113+0000","last_undegraded":"2024-04-25T01:23:27.830113+0000","last_fullsized":"2024-04-25T01:23:27.830113+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830687+0000","last_change":"2024-04-25T01:23:27.830687+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830687+0000","last_undegraded":"2024-04-25T01:23:27.830687+0000","last_fullsized":"2024-04-25T01:23:27.830687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828146+0000","last_change":"2024-04-25T01:23:27.828146+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828146+0000","last_undegraded":"2024-04-25T01:23:27.828146+0000","last_fullsized":"2024-04-25T01:23:27.828146+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838894+0000","last_change":"2024-04-25T01:23:27.838894+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838894+0000","last_undegraded":"2024-04-25T01:23:27.838894+0000","last_fullsized":"2024-04-25T01:23:27.838894+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834175+0000","last_change":"2024-04-25T01:23:27.834175+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834175+0000","last_undegraded":"2024-04-25T01:23:27.834175+0000","last_fullsized":"2024-04-25T01:23:27.834175+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836796+0000","last_change":"2024-04-25T01:23:27.836796+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836796+0000","last_undegraded":"2024-04-25T01:23:27.836796+0000","last_fullsized":"2024-04-25T01:23:27.836796+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827603+0000","last_change":"2024-04-25T01:23:27.827603+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827603+0000","last_undegraded":"2024-04-25T01:23:27.827603+0000","last_fullsized":"2024-04-25T01:23:27.827603+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830254+0000","last_change":"2024-04-25T01:23:27.830254+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830254+0000","last_undegraded":"2024-04-25T01:23:27.830254+0000","last_fullsized":"2024-04-25T01:23:27.830254+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833652+0000","last_change":"2024-04-25T01:23:27.833652+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833652+0000","last_undegraded":"2024-04-25T01:23:27.833652+0000","last_fullsized":"2024-04-25T01:23:27.833652+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827511+0000","last_change":"2024-04-25T01:23:27.827511+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827511+0000","last_undegraded":"2024-04-25T01:23:27.827511+0000","last_fullsized":"2024-04-25T01:23:27.827511+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838536+0000","last_change":"2024-04-25T01:23:27.838536+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838536+0000","last_undegraded":"2024-04-25T01:23:27.838536+0000","last_fullsized":"2024-04-25T01:23:27.838536+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.829534+0000","last_change":"2024-04-25T01:23:27.829534+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.829534+0000","last_undegraded":"2024-04-25T01:23:27.829534+0000","last_fullsized":"2024-04-25T01:23:27.829534+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.825966+0000","last_change":"2024-04-25T01:23:27.825966+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.825966+0000","last_undegraded":"2024-04-25T01:23:27.825966+0000","last_fullsized":"2024-04-25T01:23:27.825966+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838142+0000","last_change":"2024-04-25T01:23:27.838142+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838142+0000","last_undegraded":"2024-04-25T01:23:27.838142+0000","last_fullsized":"2024-04-25T01:23:27.838142+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832007+0000","last_change":"2024-04-25T01:23:27.832007+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832007+0000","last_undegraded":"2024-04-25T01:23:27.832007+0000","last_fullsized":"2024-04-25T01:23:27.832007+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836983+0000","last_change":"2024-04-25T01:23:27.836983+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836983+0000","last_undegraded":"2024-04-25T01:23:27.836983+0000","last_fullsized":"2024-04-25T01:23:27.836983+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832039+0000","last_change":"2024-04-25T01:23:27.832039+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832039+0000","last_undegraded":"2024-04-25T01:23:27.832039+0000","last_fullsized":"2024-04-25T01:23:27.832039+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830258+0000","last_change":"2024-04-25T01:23:27.830258+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830258+0000","last_undegraded":"2024-04-25T01:23:27.830258+0000","last_fullsized":"2024-04-25T01:23:27.830258+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833623+0000","last_change":"2024-04-25T01:23:27.833623+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833623+0000","last_undegraded":"2024-04-25T01:23:27.833623+0000","last_fullsized":"2024-04-25T01:23:27.833623+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830382+0000","last_change":"2024-04-25T01:23:27.830382+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830382+0000","last_undegraded":"2024-04-25T01:23:27.830382+0000","last_fullsized":"2024-04-25T01:23:27.830382+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836595+0000","last_change":"2024-04-25T01:23:27.836595+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836595+0000","last_undegraded":"2024-04-25T01:23:27.836595+0000","last_fullsized":"2024-04-25T01:23:27.836595+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836402+0000","last_change":"2024-04-25T01:23:27.836402+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836402+0000","last_undegraded":"2024-04-25T01:23:27.836402+0000","last_fullsized":"2024-04-25T01:23:27.836402+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835991+0000","last_change":"2024-04-25T01:23:27.835991+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835991+0000","last_undegraded":"2024-04-25T01:23:27.835991+0000","last_fullsized":"2024-04-25T01:23:27.835991+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827570+0000","last_change":"2024-04-25T01:23:27.827570+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827570+0000","last_undegraded":"2024-04-25T01:23:27.827570+0000","last_fullsized":"2024-04-25T01:23:27.827570+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831597+0000","last_change":"2024-04-25T01:23:27.831597+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831597+0000","last_undegraded":"2024-04-25T01:23:27.831597+0000","last_fullsized":"2024-04-25T01:23:27.831597+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836573+0000","last_change":"2024-04-25T01:23:27.836573+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836573+0000","last_undegraded":"2024-04-25T01:23:27.836573+0000","last_fullsized":"2024-04-25T01:23:27.836573+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830474+0000","last_change":"2024-04-25T01:23:27.830474+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830474+0000","last_undegraded":"2024-04-25T01:23:27.830474+0000","last_fullsized":"2024-04-25T01:23:27.830474+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835732+0000","last_change":"2024-04-25T01:23:27.835732+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835732+0000","last_undegraded":"2024-04-25T01:23:27.835732+0000","last_fullsized":"2024-04-25T01:23:27.835732+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828172+0000","last_change":"2024-04-25T01:23:27.828172+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828172+0000","last_undegraded":"2024-04-25T01:23:27.828172+0000","last_fullsized":"2024-04-25T01:23:27.828172+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830914+0000","last_change":"2024-04-25T01:23:27.830914+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830914+0000","last_undegraded":"2024-04-25T01:23:27.830914+0000","last_fullsized":"2024-04-25T01:23:27.830914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844780+0000","last_change":"2024-04-25T01:23:27.840369+0000","last_active":"2024-04-25T01:23:27.844780+0000","last_peered":"2024-04-25T01:23:27.844780+0000","last_clean":"2024-04-25T01:23:27.844780+0000","last_became_active":"2024-04-25T01:23:27.839987+0000","last_became_peered":"2024-04-25T01:23:27.839987+0000","last_unstale":"2024-04-25T01:23:27.844780+0000","last_undegraded":"2024-04-25T01:23:27.844780+0000","last_fullsized":"2024-04-25T01:23:27.844780+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.219905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826648+0000","last_change":"2024-04-25T01:23:27.826648+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826648+0000","last_undegraded":"2024-04-25T01:23:27.826648+0000","last_fullsized":"2024-04-25T01:23:27.826648+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.841990+0000","last_change":"2024-04-25T01:23:27.838394+0000","last_active":"2024-04-25T01:23:27.841990+0000","last_peered":"2024-04-25T01:23:27.841990+0000","last_clean":"2024-04-25T01:23:27.841990+0000","last_became_active":"2024-04-25T01:23:27.838114+0000","last_became_peered":"2024-04-25T01:23:27.838114+0000","last_unstale":"2024-04-25T01:23:27.841990+0000","last_undegraded":"2024-04-25T01:23:27.841990+0000","last_fullsized":"2024-04-25T01:23:27.841990+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:48.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832822+0000","last_change":"2024-04-25T01:23:27.832822+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832822+0000","last_undegraded":"2024-04-25T01:23:27.832822+0000","last_fullsized":"2024-04-25T01:23:27.832822+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.842100+0000","last_change":"2024-04-25T01:23:27.839860+0000","last_active":"2024-04-25T01:23:27.842100+0000","last_peered":"2024-04-25T01:23:27.842100+0000","last_clean":"2024-04-25T01:23:27.842100+0000","last_became_active":"2024-04-25T01:23:27.839589+0000","last_became_peered":"2024-04-25T01:23:27.839589+0000","last_unstale":"2024-04-25T01:23:27.842100+0000","last_undegraded":"2024-04-25T01:23:27.842100+0000","last_fullsized":"2024-04-25T01:23:27.842100+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:52.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835561+0000","last_change":"2024-04-25T01:23:27.835561+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835561+0000","last_undegraded":"2024-04-25T01:23:27.835561+0000","last_fullsized":"2024-04-25T01:23:27.835561+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847649+0000","last_change":"2024-04-25T01:23:27.842142+0000","last_active":"2024-04-25T01:23:27.847649+0000","last_peered":"2024-04-25T01:23:27.847649+0000","last_clean":"2024-04-25T01:23:27.847649+0000","last_became_active":"2024-04-25T01:23:27.841740+0000","last_became_peered":"2024-04-25T01:23:27.841740+0000","last_unstale":"2024-04-25T01:23:27.847649+0000","last_undegraded":"2024-04-25T01:23:27.847649+0000","last_fullsized":"2024-04-25T01:23:27.847649+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:13.292186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831025+0000","last_change":"2024-04-25T01:23:27.831025+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831025+0000","last_undegraded":"2024-04-25T01:23:27.831025+0000","last_fullsized":"2024-04-25T01:23:27.831025+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844533+0000","last_change":"2024-04-25T01:23:27.831214+0000","last_active":"2024-04-25T01:23:27.844533+0000","last_peered":"2024-04-25T01:23:27.844533+0000","last_clean":"2024-04-25T01:23:27.844533+0000","last_became_active":"2024-04-25T01:23:27.830883+0000","last_became_peered":"2024-04-25T01:23:27.830883+0000","last_unstale":"2024-04-25T01:23:27.844533+0000","last_undegraded":"2024-04-25T01:23:27.844533+0000","last_fullsized":"2024-04-25T01:23:27.844533+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:40.061850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830940+0000","last_change":"2024-04-25T01:23:27.830940+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830940+0000","last_undegraded":"2024-04-25T01:23:27.830940+0000","last_fullsized":"2024-04-25T01:23:27.830940+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844655+0000","last_change":"2024-04-25T01:23:27.839506+0000","last_active":"2024-04-25T01:23:27.844655+0000","last_peered":"2024-04-25T01:23:27.844655+0000","last_clean":"2024-04-25T01:23:27.844655+0000","last_became_active":"2024-04-25T01:23:27.838600+0000","last_became_peered":"2024-04-25T01:23:27.838600+0000","last_unstale":"2024-04-25T01:23:27.844655+0000","last_undegraded":"2024-04-25T01:23:27.844655+0000","last_fullsized":"2024-04-25T01:23:27.844655+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:48.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833846+0000","last_change":"2024-04-25T01:23:27.833846+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833846+0000","last_undegraded":"2024-04-25T01:23:27.833846+0000","last_fullsized":"2024-04-25T01:23:27.833846+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847468+0000","last_change":"2024-04-25T01:23:27.843276+0000","last_active":"2024-04-25T01:23:27.847468+0000","last_peered":"2024-04-25T01:23:27.847468+0000","last_clean":"2024-04-25T01:23:27.847468+0000","last_became_active":"2024-04-25T01:23:27.842824+0000","last_became_peered":"2024-04-25T01:23:27.842824+0000","last_unstale":"2024-04-25T01:23:27.847468+0000","last_undegraded":"2024-04-25T01:23:27.847468+0000","last_fullsized":"2024-04-25T01:23:27.847468+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:48.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831669+0000","last_change":"2024-04-25T01:23:27.831669+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831669+0000","last_undegraded":"2024-04-25T01:23:27.831669+0000","last_fullsized":"2024-04-25T01:23:27.831669+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848229+0000","last_change":"2024-04-25T01:23:27.842530+0000","last_active":"2024-04-25T01:23:27.848229+0000","last_peered":"2024-04-25T01:23:27.848229+0000","last_clean":"2024-04-25T01:23:27.848229+0000","last_became_active":"2024-04-25T01:23:27.842265+0000","last_became_peered":"2024-04-25T01:23:27.842265+0000","last_unstale":"2024-04-25T01:23:27.848229+0000","last_undegraded":"2024-04-25T01:23:27.848229+0000","last_fullsized":"2024-04-25T01:23:27.848229+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:49:28.578991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.829914+0000","last_change":"2024-04-25T01:23:27.829914+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.829914+0000","last_undegraded":"2024-04-25T01:23:27.829914+0000","last_fullsized":"2024-04-25T01:23:27.829914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847439+0000","last_change":"2024-04-25T01:23:27.842714+0000","last_active":"2024-04-25T01:23:27.847439+0000","last_peered":"2024-04-25T01:23:27.847439+0000","last_clean":"2024-04-25T01:23:27.847439+0000","last_became_active":"2024-04-25T01:23:27.842268+0000","last_became_peered":"2024-04-25T01:23:27.842268+0000","last_unstale":"2024-04-25T01:23:27.847439+0000","last_undegraded":"2024-04-25T01:23:27.847439+0000","last_fullsized":"2024-04-25T01:23:27.847439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:01.477667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827321+0000","last_change":"2024-04-25T01:23:27.827321+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827321+0000","last_undegraded":"2024-04-25T01:23:27.827321+0000","last_fullsized":"2024-04-25T01:23:27.827321+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844619+0000","last_change":"2024-04-25T01:23:27.839254+0000","last_active":"2024-04-25T01:23:27.844619+0000","last_peered":"2024-04-25T01:23:27.844619+0000","last_clean":"2024-04-25T01:23:27.844619+0000","last_became_active":"2024-04-25T01:23:27.838972+0000","last_became_peered":"2024-04-25T01:23:27.838972+0000","last_unstale":"2024-04-25T01:23:27.844619+0000","last_undegraded":"2024-04-25T01:23:27.844619+0000","last_fullsized":"2024-04-25T01:23:27.844619+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:04.719084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834497+0000","last_change":"2024-04-25T01:23:27.834497+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834497+0000","last_undegraded":"2024-04-25T01:23:27.834497+0000","last_fullsized":"2024-04-25T01:23:27.834497+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847498+0000","last_change":"2024-04-25T01:23:27.843358+0000","last_active":"2024-04-25T01:23:27.847498+0000","last_peered":"2024-04-25T01:23:27.847498+0000","last_clean":"2024-04-25T01:23:27.847498+0000","last_became_active":"2024-04-25T01:23:27.843098+0000","last_became_peered":"2024-04-25T01:23:27.843098+0000","last_unstale":"2024-04-25T01:23:27.847498+0000","last_undegraded":"2024-04-25T01:23:27.847498+0000","last_fullsized":"2024-04-25T01:23:27.847498+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:38.211801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830273+0000","last_change":"2024-04-25T01:23:27.830273+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830273+0000","last_undegraded":"2024-04-25T01:23:27.830273+0000","last_fullsized":"2024-04-25T01:23:27.830273+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846301+0000","last_change":"2024-04-25T01:23:27.841668+0000","last_active":"2024-04-25T01:23:27.846301+0000","last_peered":"2024-04-25T01:23:27.846301+0000","last_clean":"2024-04-25T01:23:27.846301+0000","last_became_active":"2024-04-25T01:23:27.841382+0000","last_became_peered":"2024-04-25T01:23:27.841382+0000","last_unstale":"2024-04-25T01:23:27.846301+0000","last_undegraded":"2024-04-25T01:23:27.846301+0000","last_fullsized":"2024-04-25T01:23:27.846301+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:00:11.736192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827805+0000","last_change":"2024-04-25T01:23:27.827805+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827805+0000","last_undegraded":"2024-04-25T01:23:27.827805+0000","last_fullsized":"2024-04-25T01:23:27.827805+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848483+0000","last_change":"2024-04-25T01:23:27.845237+0000","last_active":"2024-04-25T01:23:27.848483+0000","last_peered":"2024-04-25T01:23:27.848483+0000","last_clean":"2024-04-25T01:23:27.848483+0000","last_became_active":"2024-04-25T01:23:27.844336+0000","last_became_peered":"2024-04-25T01:23:27.844336+0000","last_unstale":"2024-04-25T01:23:27.848483+0000","last_undegraded":"2024-04-25T01:23:27.848483+0000","last_fullsized":"2024-04-25T01:23:27.848483+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:52.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834532+0000","last_change":"2024-04-25T01:23:27.834532+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834532+0000","last_undegraded":"2024-04-25T01:23:27.834532+0000","last_fullsized":"2024-04-25T01:23:27.834532+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.849509+0000","last_change":"2024-04-25T01:23:27.848962+0000","last_active":"2024-04-25T01:23:27.849509+0000","last_peered":"2024-04-25T01:23:27.849509+0000","last_clean":"2024-04-25T01:23:27.849509+0000","last_became_active":"2024-04-25T01:23:27.848694+0000","last_became_peered":"2024-04-25T01:23:27.848694+0000","last_unstale":"2024-04-25T01:23:27.849509+0000","last_undegraded":"2024-04-25T01:23:27.849509+0000","last_fullsized":"2024-04-25T01:23:27.849509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:04.282731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834242+0000","last_change":"2024-04-25T01:23:27.834242+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834242+0000","last_undegraded":"2024-04-25T01:23:27.834242+0000","last_fullsized":"2024-04-25T01:23:27.834242+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847532+0000","last_change":"2024-04-25T01:23:27.843254+0000","last_active":"2024-04-25T01:23:27.847532+0000","last_peered":"2024-04-25T01:23:27.847532+0000","last_clean":"2024-04-25T01:23:27.847532+0000","last_became_active":"2024-04-25T01:23:27.842506+0000","last_became_peered":"2024-04-25T01:23:27.842506+0000","last_unstale":"2024-04-25T01:23:27.847532+0000","last_undegraded":"2024-04-25T01:23:27.847532+0000","last_fullsized":"2024-04-25T01:23:27.847532+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:42.248852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835026+0000","last_change":"2024-04-25T01:23:27.835026+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835026+0000","last_undegraded":"2024-04-25T01:23:27.835026+0000","last_fullsized":"2024-04-25T01:23:27.835026+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846277+0000","last_change":"2024-04-25T01:23:27.841058+0000","last_active":"2024-04-25T01:23:27.846277+0000","last_peered":"2024-04-25T01:23:27.846277+0000","last_clean":"2024-04-25T01:23:27.846277+0000","last_became_active":"2024-04-25T01:23:27.840808+0000","last_became_peered":"2024-04-25T01:23:27.840808+0000","last_unstale":"2024-04-25T01:23:27.846277+0000","last_undegraded":"2024-04-25T01:23:27.846277+0000","last_fullsized":"2024-04-25T01:23:27.846277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:03.437919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833035+0000","last_change":"2024-04-25T01:23:27.833035+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833035+0000","last_undegraded":"2024-04-25T01:23:27.833035+0000","last_fullsized":"2024-04-25T01:23:27.833035+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848509+0000","last_change":"2024-04-25T01:23:27.846019+0000","last_active":"2024-04-25T01:23:27.848509+0000","last_peered":"2024-04-25T01:23:27.848509+0000","last_clean":"2024-04-25T01:23:27.848509+0000","last_became_active":"2024-04-25T01:23:27.845056+0000","last_became_peered":"2024-04-25T01:23:27.845056+0000","last_unstale":"2024-04-25T01:23:27.848509+0000","last_undegraded":"2024-04-25T01:23:27.848509+0000","last_fullsized":"2024-04-25T01:23:27.848509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:08.739387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826903+0000","last_change":"2024-04-25T01:23:27.826903+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826903+0000","last_undegraded":"2024-04-25T01:23:27.826903+0000","last_fullsized":"2024-04-25T01:23:27.826903+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.845756+0000","last_change":"2024-04-25T01:23:27.839989+0000","last_active":"2024-04-25T01:23:27.845756+0000","last_peered":"2024-04-25T01:23:27.845756+0000","last_clean":"2024-04-25T01:23:27.845756+0000","last_became_active":"2024-04-25T01:23:27.839737+0000","last_became_peered":"2024-04-25T01:23:27.839737+0000","last_unstale":"2024-04-25T01:23:27.845756+0000","last_undegraded":"2024-04-25T01:23:27.845756+0000","last_fullsized":"2024-04-25T01:23:27.845756+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:17.100820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830591+0000","last_change":"2024-04-25T01:23:27.830591+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830591+0000","last_undegraded":"2024-04-25T01:23:27.830591+0000","last_fullsized":"2024-04-25T01:23:27.830591+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.845062+0000","last_change":"2024-04-25T01:23:27.838563+0000","last_active":"2024-04-25T01:23:27.845062+0000","last_peered":"2024-04-25T01:23:27.845062+0000","last_clean":"2024-04-25T01:23:27.845062+0000","last_became_active":"2024-04-25T01:23:27.837764+0000","last_became_peered":"2024-04-25T01:23:27.837764+0000","last_unstale":"2024-04-25T01:23:27.845062+0000","last_undegraded":"2024-04-25T01:23:27.845062+0000","last_fullsized":"2024-04-25T01:23:27.845062+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:26.874008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826397+0000","last_change":"2024-04-25T01:23:27.826397+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826397+0000","last_undegraded":"2024-04-25T01:23:27.826397+0000","last_fullsized":"2024-04-25T01:23:27.826397+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847457+0000","last_change":"2024-04-25T01:23:27.843101+0000","last_active":"2024-04-25T01:23:27.847457+0000","last_peered":"2024-04-25T01:23:27.847457+0000","last_clean":"2024-04-25T01:23:27.847457+0000","last_became_active":"2024-04-25T01:23:27.842810+0000","last_became_peered":"2024-04-25T01:23:27.842810+0000","last_unstale":"2024-04-25T01:23:27.847457+0000","last_undegraded":"2024-04-25T01:23:27.847457+0000","last_fullsized":"2024-04-25T01:23:27.847457+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:04.719084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826783+0000","last_change":"2024-04-25T01:23:27.826783+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826783+0000","last_undegraded":"2024-04-25T01:23:27.826783+0000","last_fullsized":"2024-04-25T01:23:27.826783+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.838395+0000","last_change":"2024-04-25T01:23:27.834285+0000","last_active":"2024-04-25T01:23:27.838395+0000","last_peered":"2024-04-25T01:23:27.838395+0000","last_clean":"2024-04-25T01:23:27.838395+0000","last_became_active":"2024-04-25T01:23:27.834030+0000","last_became_peered":"2024-04-25T01:23:27.834030+0000","last_unstale":"2024-04-25T01:23:27.838395+0000","last_undegraded":"2024-04-25T01:23:27.838395+0000","last_fullsized":"2024-04-25T01:23:27.838395+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:33.027603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832642+0000","last_change":"2024-04-25T01:23:27.832642+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832642+0000","last_undegraded":"2024-04-25T01:23:27.832642+0000","last_fullsized":"2024-04-25T01:23:27.832642+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846225+0000","last_change":"2024-04-25T01:23:27.839744+0000","last_active":"2024-04-25T01:23:27.846225+0000","last_peered":"2024-04-25T01:23:27.846225+0000","last_clean":"2024-04-25T01:23:27.846225+0000","last_became_active":"2024-04-25T01:23:27.839478+0000","last_became_peered":"2024-04-25T01:23:27.839478+0000","last_unstale":"2024-04-25T01:23:27.846225+0000","last_undegraded":"2024-04-25T01:23:27.846225+0000","last_fullsized":"2024-04-25T01:23:27.846225+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:44.264978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833162+0000","last_change":"2024-04-25T01:23:27.833162+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833162+0000","last_undegraded":"2024-04-25T01:23:27.833162+0000","last_fullsized":"2024-04-25T01:23:27.833162+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848293+0000","last_change":"2024-04-25T01:23:27.843276+0000","last_active":"2024-04-25T01:23:27.848293+0000","last_peered":"2024-04-25T01:23:27.848293+0000","last_clean":"2024-04-25T01:23:27.848293+0000","last_became_active":"2024-04-25T01:23:27.842913+0000","last_became_peered":"2024-04-25T01:23:27.842913+0000","last_unstale":"2024-04-25T01:23:27.848293+0000","last_undegraded":"2024-04-25T01:23:27.848293+0000","last_fullsized":"2024-04-25T01:23:27.848293+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:33.027603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826481+0000","last_change":"2024-04-25T01:23:27.826481+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826481+0000","last_undegraded":"2024-04-25T01:23:27.826481+0000","last_fullsized":"2024-04-25T01:23:27.826481+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846474+0000","last_change":"2024-04-25T01:23:27.843723+0000","last_active":"2024-04-25T01:23:27.846474+0000","last_peered":"2024-04-25T01:23:27.846474+0000","last_clean":"2024-04-25T01:23:27.846474+0000","last_became_active":"2024-04-25T01:23:27.843464+0000","last_became_peered":"2024-04-25T01:23:27.843464+0000","last_unstale":"2024-04-25T01:23:27.846474+0000","last_undegraded":"2024-04-25T01:23:27.846474+0000","last_fullsized":"2024-04-25T01:23:27.846474+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:01.926860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826655+0000","last_change":"2024-04-25T01:23:27.826655+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826655+0000","last_undegraded":"2024-04-25T01:23:27.826655+0000","last_fullsized":"2024-04-25T01:23:27.826655+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848098+0000","last_change":"2024-04-25T01:23:27.847328+0000","last_active":"2024-04-25T01:23:27.848098+0000","last_peered":"2024-04-25T01:23:27.848098+0000","last_clean":"2024-04-25T01:23:27.848098+0000","last_became_active":"2024-04-25T01:23:27.847046+0000","last_became_peered":"2024-04-25T01:23:27.847046+0000","last_unstale":"2024-04-25T01:23:27.848098+0000","last_undegraded":"2024-04-25T01:23:27.848098+0000","last_fullsized":"2024-04-25T01:23:27.848098+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:59.145724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827287+0000","last_change":"2024-04-25T01:23:27.827287+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827287+0000","last_undegraded":"2024-04-25T01:23:27.827287+0000","last_fullsized":"2024-04-25T01:23:27.827287+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847476+0000","last_change":"2024-04-25T01:23:27.842809+0000","last_active":"2024-04-25T01:23:27.847476+0000","last_peered":"2024-04-25T01:23:27.847476+0000","last_clean":"2024-04-25T01:23:27.847476+0000","last_became_active":"2024-04-25T01:23:27.842561+0000","last_became_peered":"2024-04-25T01:23:27.842561+0000","last_unstale":"2024-04-25T01:23:27.847476+0000","last_undegraded":"2024-04-25T01:23:27.847476+0000","last_fullsized":"2024-04-25T01:23:27.847476+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:28:34.927620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835126+0000","last_change":"2024-04-25T01:23:27.835126+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835126+0000","last_undegraded":"2024-04-25T01:23:27.835126+0000","last_fullsized":"2024-04-25T01:23:27.835126+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848367+0000","last_change":"2024-04-25T01:23:27.844808+0000","last_active":"2024-04-25T01:23:27.848367+0000","last_peered":"2024-04-25T01:23:27.848367+0000","last_clean":"2024-04-25T01:23:27.848367+0000","last_became_active":"2024-04-25T01:23:27.844541+0000","last_became_peered":"2024-04-25T01:23:27.844541+0000","last_unstale":"2024-04-25T01:23:27.848367+0000","last_undegraded":"2024-04-25T01:23:27.848367+0000","last_fullsized":"2024-04-25T01:23:27.848367+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:48.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832949+0000","last_change":"2024-04-25T01:23:27.832949+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832949+0000","last_undegraded":"2024-04-25T01:23:27.832949+0000","last_fullsized":"2024-04-25T01:23:27.832949+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846394+0000","last_change":"2024-04-25T01:23:27.842454+0000","last_active":"2024-04-25T01:23:27.846394+0000","last_peered":"2024-04-25T01:23:27.846394+0000","last_clean":"2024-04-25T01:23:27.846394+0000","last_became_active":"2024-04-25T01:23:27.842209+0000","last_became_peered":"2024-04-25T01:23:27.842209+0000","last_unstale":"2024-04-25T01:23:27.846394+0000","last_undegraded":"2024-04-25T01:23:27.846394+0000","last_fullsized":"2024-04-25T01:23:27.846394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:22.705548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826687+0000","last_change":"2024-04-25T01:23:27.826687+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826687+0000","last_undegraded":"2024-04-25T01:23:27.826687+0000","last_fullsized":"2024-04-25T01:23:27.826687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846248+0000","last_change":"2024-04-25T01:23:27.841080+0000","last_active":"2024-04-25T01:23:27.846248+0000","last_peered":"2024-04-25T01:23:27.846248+0000","last_clean":"2024-04-25T01:23:27.846248+0000","last_became_active":"2024-04-25T01:23:27.840813+0000","last_became_peered":"2024-04-25T01:23:27.840813+0000","last_unstale":"2024-04-25T01:23:27.846248+0000","last_undegraded":"2024-04-25T01:23:27.846248+0000","last_fullsized":"2024-04-25T01:23:27.846248+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:44.720263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833838+0000","last_change":"2024-04-25T01:23:27.833838+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833838+0000","last_undegraded":"2024-04-25T01:23:27.833838+0000","last_fullsized":"2024-04-25T01:23:27.833838+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844601+0000","last_change":"2024-04-25T01:23:27.841167+0000","last_active":"2024-04-25T01:23:27.844601+0000","last_peered":"2024-04-25T01:23:27.844601+0000","last_clean":"2024-04-25T01:23:27.844601+0000","last_became_active":"2024-04-25T01:23:27.840932+0000","last_became_peered":"2024-04-25T01:23:27.840932+0000","last_unstale":"2024-04-25T01:23:27.844601+0000","last_undegraded":"2024-04-25T01:23:27.844601+0000","last_fullsized":"2024-04-25T01:23:27.844601+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:59.145724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836435+0000","last_change":"2024-04-25T01:23:27.836435+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836435+0000","last_undegraded":"2024-04-25T01:23:27.836435+0000","last_fullsized":"2024-04-25T01:23:27.836435+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.842097+0000","last_change":"2024-04-25T01:23:27.838507+0000","last_active":"2024-04-25T01:23:27.842097+0000","last_peered":"2024-04-25T01:23:27.842097+0000","last_clean":"2024-04-25T01:23:27.842097+0000","last_became_active":"2024-04-25T01:23:27.838016+0000","last_became_peered":"2024-04-25T01:23:27.838016+0000","last_unstale":"2024-04-25T01:23:27.842097+0000","last_undegraded":"2024-04-25T01:23:27.842097+0000","last_fullsized":"2024-04-25T01:23:27.842097+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:09.155513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831341+0000","last_change":"2024-04-25T01:23:27.831341+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831341+0000","last_undegraded":"2024-04-25T01:23:27.831341+0000","last_fullsized":"2024-04-25T01:23:27.831341+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846638+0000","last_change":"2024-04-25T01:23:27.841576+0000","last_active":"2024-04-25T01:23:27.846638+0000","last_peered":"2024-04-25T01:23:27.846638+0000","last_clean":"2024-04-25T01:23:27.846638+0000","last_became_active":"2024-04-25T01:23:27.839038+0000","last_became_peered":"2024-04-25T01:23:27.839038+0000","last_unstale":"2024-04-25T01:23:27.846638+0000","last_undegraded":"2024-04-25T01:23:27.846638+0000","last_fullsized":"2024-04-25T01:23:27.846638+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:26:54.800425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":222,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.833349+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:23:27.833349+0000","last_peered":"2024-04-25T01:23:27.833349+0000","last_clean":"2024-04-25T01:23:27.833349+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:23:27.833349+0000","last_undegraded":"2024-04-25T01:23:27.833349+0000","last_fullsized":"2024-04-25T01:23:27.833349+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1163264,"data_stored":840266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1163264,"data_stored":840266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:23:32.465 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3f is not active+clean 2024-04-25T01:23:32.465 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.828699+0000', 'last_change': '2024-04-25T01:23:27.828699+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.828699+0000', 'last_undegraded': '2024-04-25T01:23:27.828699+0000', 'last_fullsized': '2024-04-25T01:23:27.828699+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T01:23:32.465 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3e is not active+clean 2024-04-25T01:23:32.465 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.833069+0000', 'last_change': '2024-04-25T01:23:27.833069+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.833069+0000', 'last_undegraded': '2024-04-25T01:23:27.833069+0000', 'last_fullsized': '2024-04-25T01:23:27.833069+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T01:23:32.466 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3d is not active+clean 2024-04-25T01:23:32.466 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.838229+0000', 'last_change': '2024-04-25T01:23:27.838229+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.838229+0000', 'last_undegraded': '2024-04-25T01:23:27.838229+0000', 'last_fullsized': '2024-04-25T01:23:27.838229+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.466 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3c is not active+clean 2024-04-25T01:23:32.466 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.830113+0000', 'last_change': '2024-04-25T01:23:27.830113+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.830113+0000', 'last_undegraded': '2024-04-25T01:23:27.830113+0000', 'last_fullsized': '2024-04-25T01:23:27.830113+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T01:23:32.466 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3b is not active+clean 2024-04-25T01:23:32.466 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.830687+0000', 'last_change': '2024-04-25T01:23:27.830687+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.830687+0000', 'last_undegraded': '2024-04-25T01:23:27.830687+0000', 'last_fullsized': '2024-04-25T01:23:27.830687+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T01:23:32.466 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3a is not active+clean 2024-04-25T01:23:32.466 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.828146+0000', 'last_change': '2024-04-25T01:23:27.828146+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.828146+0000', 'last_undegraded': '2024-04-25T01:23:27.828146+0000', 'last_fullsized': '2024-04-25T01:23:27.828146+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T01:23:32.467 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.39 is not active+clean 2024-04-25T01:23:32.467 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.838894+0000', 'last_change': '2024-04-25T01:23:27.838894+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.838894+0000', 'last_undegraded': '2024-04-25T01:23:27.838894+0000', 'last_fullsized': '2024-04-25T01:23:27.838894+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.467 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.38 is not active+clean 2024-04-25T01:23:32.467 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.834175+0000', 'last_change': '2024-04-25T01:23:27.834175+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.834175+0000', 'last_undegraded': '2024-04-25T01:23:27.834175+0000', 'last_fullsized': '2024-04-25T01:23:27.834175+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T01:23:32.467 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.37 is not active+clean 2024-04-25T01:23:32.467 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.836796+0000', 'last_change': '2024-04-25T01:23:27.836796+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.836796+0000', 'last_undegraded': '2024-04-25T01:23:27.836796+0000', 'last_fullsized': '2024-04-25T01:23:27.836796+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T01:23:32.467 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.36 is not active+clean 2024-04-25T01:23:32.467 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.827603+0000', 'last_change': '2024-04-25T01:23:27.827603+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.827603+0000', 'last_undegraded': '2024-04-25T01:23:27.827603+0000', 'last_fullsized': '2024-04-25T01:23:27.827603+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T01:23:32.467 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.35 is not active+clean 2024-04-25T01:23:32.467 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.830254+0000', 'last_change': '2024-04-25T01:23:27.830254+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.830254+0000', 'last_undegraded': '2024-04-25T01:23:27.830254+0000', 'last_fullsized': '2024-04-25T01:23:27.830254+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T01:23:32.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.34 is not active+clean 2024-04-25T01:23:32.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.833652+0000', 'last_change': '2024-04-25T01:23:27.833652+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.833652+0000', 'last_undegraded': '2024-04-25T01:23:27.833652+0000', 'last_fullsized': '2024-04-25T01:23:27.833652+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T01:23:32.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.33 is not active+clean 2024-04-25T01:23:32.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.827511+0000', 'last_change': '2024-04-25T01:23:27.827511+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.827511+0000', 'last_undegraded': '2024-04-25T01:23:27.827511+0000', 'last_fullsized': '2024-04-25T01:23:27.827511+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T01:23:32.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.32 is not active+clean 2024-04-25T01:23:32.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.838536+0000', 'last_change': '2024-04-25T01:23:27.838536+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.838536+0000', 'last_undegraded': '2024-04-25T01:23:27.838536+0000', 'last_fullsized': '2024-04-25T01:23:27.838536+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.31 is not active+clean 2024-04-25T01:23:32.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.829534+0000', 'last_change': '2024-04-25T01:23:27.829534+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.829534+0000', 'last_undegraded': '2024-04-25T01:23:27.829534+0000', 'last_fullsized': '2024-04-25T01:23:27.829534+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T01:23:32.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.30 is not active+clean 2024-04-25T01:23:32.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.825966+0000', 'last_change': '2024-04-25T01:23:27.825966+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.825966+0000', 'last_undegraded': '2024-04-25T01:23:27.825966+0000', 'last_fullsized': '2024-04-25T01:23:27.825966+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T01:23:32.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2f is not active+clean 2024-04-25T01:23:32.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.838142+0000', 'last_change': '2024-04-25T01:23:27.838142+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.838142+0000', 'last_undegraded': '2024-04-25T01:23:27.838142+0000', 'last_fullsized': '2024-04-25T01:23:27.838142+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2e is not active+clean 2024-04-25T01:23:32.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.832007+0000', 'last_change': '2024-04-25T01:23:27.832007+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.832007+0000', 'last_undegraded': '2024-04-25T01:23:27.832007+0000', 'last_fullsized': '2024-04-25T01:23:27.832007+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T01:23:32.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2d is not active+clean 2024-04-25T01:23:32.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.836983+0000', 'last_change': '2024-04-25T01:23:27.836983+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.836983+0000', 'last_undegraded': '2024-04-25T01:23:27.836983+0000', 'last_fullsized': '2024-04-25T01:23:27.836983+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2c is not active+clean 2024-04-25T01:23:32.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.832039+0000', 'last_change': '2024-04-25T01:23:27.832039+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.832039+0000', 'last_undegraded': '2024-04-25T01:23:27.832039+0000', 'last_fullsized': '2024-04-25T01:23:27.832039+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T01:23:32.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2b is not active+clean 2024-04-25T01:23:32.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.830258+0000', 'last_change': '2024-04-25T01:23:27.830258+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.830258+0000', 'last_undegraded': '2024-04-25T01:23:27.830258+0000', 'last_fullsized': '2024-04-25T01:23:27.830258+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T01:23:32.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2a is not active+clean 2024-04-25T01:23:32.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.833623+0000', 'last_change': '2024-04-25T01:23:27.833623+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.833623+0000', 'last_undegraded': '2024-04-25T01:23:27.833623+0000', 'last_fullsized': '2024-04-25T01:23:27.833623+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T01:23:32.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.29 is not active+clean 2024-04-25T01:23:32.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.830382+0000', 'last_change': '2024-04-25T01:23:27.830382+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.830382+0000', 'last_undegraded': '2024-04-25T01:23:27.830382+0000', 'last_fullsized': '2024-04-25T01:23:27.830382+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T01:23:32.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.28 is not active+clean 2024-04-25T01:23:32.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.836595+0000', 'last_change': '2024-04-25T01:23:27.836595+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.836595+0000', 'last_undegraded': '2024-04-25T01:23:27.836595+0000', 'last_fullsized': '2024-04-25T01:23:27.836595+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.27 is not active+clean 2024-04-25T01:23:32.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.836402+0000', 'last_change': '2024-04-25T01:23:27.836402+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.836402+0000', 'last_undegraded': '2024-04-25T01:23:27.836402+0000', 'last_fullsized': '2024-04-25T01:23:27.836402+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.26 is not active+clean 2024-04-25T01:23:32.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.835991+0000', 'last_change': '2024-04-25T01:23:27.835991+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.835991+0000', 'last_undegraded': '2024-04-25T01:23:27.835991+0000', 'last_fullsized': '2024-04-25T01:23:27.835991+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T01:23:32.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.25 is not active+clean 2024-04-25T01:23:32.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.827570+0000', 'last_change': '2024-04-25T01:23:27.827570+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.827570+0000', 'last_undegraded': '2024-04-25T01:23:27.827570+0000', 'last_fullsized': '2024-04-25T01:23:27.827570+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T01:23:32.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.24 is not active+clean 2024-04-25T01:23:32.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.831597+0000', 'last_change': '2024-04-25T01:23:27.831597+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.831597+0000', 'last_undegraded': '2024-04-25T01:23:27.831597+0000', 'last_fullsized': '2024-04-25T01:23:27.831597+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T01:23:32.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.23 is not active+clean 2024-04-25T01:23:32.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.836573+0000', 'last_change': '2024-04-25T01:23:27.836573+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.836573+0000', 'last_undegraded': '2024-04-25T01:23:27.836573+0000', 'last_fullsized': '2024-04-25T01:23:27.836573+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.22 is not active+clean 2024-04-25T01:23:32.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.830474+0000', 'last_change': '2024-04-25T01:23:27.830474+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.830474+0000', 'last_undegraded': '2024-04-25T01:23:27.830474+0000', 'last_fullsized': '2024-04-25T01:23:27.830474+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T01:23:32.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.21 is not active+clean 2024-04-25T01:23:32.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.835732+0000', 'last_change': '2024-04-25T01:23:27.835732+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.835732+0000', 'last_undegraded': '2024-04-25T01:23:27.835732+0000', 'last_fullsized': '2024-04-25T01:23:27.835732+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.20 is not active+clean 2024-04-25T01:23:32.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.828172+0000', 'last_change': '2024-04-25T01:23:27.828172+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.828172+0000', 'last_undegraded': '2024-04-25T01:23:27.828172+0000', 'last_fullsized': '2024-04-25T01:23:27.828172+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T01:23:32.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.10 is not active+clean 2024-04-25T01:23:32.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.830914+0000', 'last_change': '2024-04-25T01:23:27.830914+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.830914+0000', 'last_undegraded': '2024-04-25T01:23:27.830914+0000', 'last_fullsized': '2024-04-25T01:23:27.830914+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T01:23:32.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-04-25T01:23:32.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.826648+0000', 'last_change': '2024-04-25T01:23:27.826648+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.826648+0000', 'last_undegraded': '2024-04-25T01:23:27.826648+0000', 'last_fullsized': '2024-04-25T01:23:27.826648+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T01:23:32.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.12 is not active+clean 2024-04-25T01:23:32.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.832822+0000', 'last_change': '2024-04-25T01:23:27.832822+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.832822+0000', 'last_undegraded': '2024-04-25T01:23:27.832822+0000', 'last_fullsized': '2024-04-25T01:23:27.832822+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T01:23:32.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-04-25T01:23:32.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.835561+0000', 'last_change': '2024-04-25T01:23:27.835561+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.835561+0000', 'last_undegraded': '2024-04-25T01:23:27.835561+0000', 'last_fullsized': '2024-04-25T01:23:27.835561+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T01:23:32.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-04-25T01:23:32.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.831025+0000', 'last_change': '2024-04-25T01:23:27.831025+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.831025+0000', 'last_undegraded': '2024-04-25T01:23:27.831025+0000', 'last_fullsized': '2024-04-25T01:23:27.831025+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T01:23:32.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.15 is not active+clean 2024-04-25T01:23:32.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.830940+0000', 'last_change': '2024-04-25T01:23:27.830940+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.830940+0000', 'last_undegraded': '2024-04-25T01:23:27.830940+0000', 'last_fullsized': '2024-04-25T01:23:27.830940+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T01:23:32.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-04-25T01:23:32.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.833846+0000', 'last_change': '2024-04-25T01:23:27.833846+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.833846+0000', 'last_undegraded': '2024-04-25T01:23:27.833846+0000', 'last_fullsized': '2024-04-25T01:23:27.833846+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T01:23:32.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-04-25T01:23:32.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.831669+0000', 'last_change': '2024-04-25T01:23:27.831669+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.831669+0000', 'last_undegraded': '2024-04-25T01:23:27.831669+0000', 'last_fullsized': '2024-04-25T01:23:27.831669+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T01:23:32.473 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.18 is not active+clean 2024-04-25T01:23:32.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.829914+0000', 'last_change': '2024-04-25T01:23:27.829914+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.829914+0000', 'last_undegraded': '2024-04-25T01:23:27.829914+0000', 'last_fullsized': '2024-04-25T01:23:27.829914+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T01:23:32.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-04-25T01:23:32.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.827321+0000', 'last_change': '2024-04-25T01:23:27.827321+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.827321+0000', 'last_undegraded': '2024-04-25T01:23:27.827321+0000', 'last_fullsized': '2024-04-25T01:23:27.827321+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T01:23:32.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-04-25T01:23:32.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.834497+0000', 'last_change': '2024-04-25T01:23:27.834497+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.834497+0000', 'last_undegraded': '2024-04-25T01:23:27.834497+0000', 'last_fullsized': '2024-04-25T01:23:27.834497+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1b is not active+clean 2024-04-25T01:23:32.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.830273+0000', 'last_change': '2024-04-25T01:23:27.830273+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.830273+0000', 'last_undegraded': '2024-04-25T01:23:27.830273+0000', 'last_fullsized': '2024-04-25T01:23:27.830273+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T01:23:32.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-04-25T01:23:32.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.827805+0000', 'last_change': '2024-04-25T01:23:27.827805+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.827805+0000', 'last_undegraded': '2024-04-25T01:23:27.827805+0000', 'last_fullsized': '2024-04-25T01:23:27.827805+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T01:23:32.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-04-25T01:23:32.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.834532+0000', 'last_change': '2024-04-25T01:23:27.834532+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.834532+0000', 'last_undegraded': '2024-04-25T01:23:27.834532+0000', 'last_fullsized': '2024-04-25T01:23:27.834532+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-04-25T01:23:32.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.834242+0000', 'last_change': '2024-04-25T01:23:27.834242+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.834242+0000', 'last_undegraded': '2024-04-25T01:23:27.834242+0000', 'last_fullsized': '2024-04-25T01:23:27.834242+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T01:23:32.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-04-25T01:23:32.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.835026+0000', 'last_change': '2024-04-25T01:23:27.835026+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.835026+0000', 'last_undegraded': '2024-04-25T01:23:27.835026+0000', 'last_fullsized': '2024-04-25T01:23:27.835026+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-04-25T01:23:32.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.833035+0000', 'last_change': '2024-04-25T01:23:27.833035+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.833035+0000', 'last_undegraded': '2024-04-25T01:23:27.833035+0000', 'last_fullsized': '2024-04-25T01:23:27.833035+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T01:23:32.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1 is not active+clean 2024-04-25T01:23:32.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.826903+0000', 'last_change': '2024-04-25T01:23:27.826903+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.826903+0000', 'last_undegraded': '2024-04-25T01:23:27.826903+0000', 'last_fullsized': '2024-04-25T01:23:27.826903+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T01:23:32.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-04-25T01:23:32.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.830591+0000', 'last_change': '2024-04-25T01:23:27.830591+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.830591+0000', 'last_undegraded': '2024-04-25T01:23:27.830591+0000', 'last_fullsized': '2024-04-25T01:23:27.830591+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T01:23:32.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3 is not active+clean 2024-04-25T01:23:32.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.826397+0000', 'last_change': '2024-04-25T01:23:27.826397+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.826397+0000', 'last_undegraded': '2024-04-25T01:23:27.826397+0000', 'last_fullsized': '2024-04-25T01:23:27.826397+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T01:23:32.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-04-25T01:23:32.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.826783+0000', 'last_change': '2024-04-25T01:23:27.826783+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.826783+0000', 'last_undegraded': '2024-04-25T01:23:27.826783+0000', 'last_fullsized': '2024-04-25T01:23:27.826783+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T01:23:32.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.5 is not active+clean 2024-04-25T01:23:32.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.832642+0000', 'last_change': '2024-04-25T01:23:27.832642+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.832642+0000', 'last_undegraded': '2024-04-25T01:23:27.832642+0000', 'last_fullsized': '2024-04-25T01:23:27.832642+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T01:23:32.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-04-25T01:23:32.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.833162+0000', 'last_change': '2024-04-25T01:23:27.833162+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.833162+0000', 'last_undegraded': '2024-04-25T01:23:27.833162+0000', 'last_fullsized': '2024-04-25T01:23:27.833162+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.7 is not active+clean 2024-04-25T01:23:32.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.826481+0000', 'last_change': '2024-04-25T01:23:27.826481+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.826481+0000', 'last_undegraded': '2024-04-25T01:23:27.826481+0000', 'last_fullsized': '2024-04-25T01:23:27.826481+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T01:23:32.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-04-25T01:23:32.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.826655+0000', 'last_change': '2024-04-25T01:23:27.826655+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.826655+0000', 'last_undegraded': '2024-04-25T01:23:27.826655+0000', 'last_fullsized': '2024-04-25T01:23:27.826655+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T01:23:32.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.a is not active+clean 2024-04-25T01:23:32.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.827287+0000', 'last_change': '2024-04-25T01:23:27.827287+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.827287+0000', 'last_undegraded': '2024-04-25T01:23:27.827287+0000', 'last_fullsized': '2024-04-25T01:23:27.827287+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T01:23:32.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-04-25T01:23:32.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.835126+0000', 'last_change': '2024-04-25T01:23:27.835126+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.835126+0000', 'last_undegraded': '2024-04-25T01:23:27.835126+0000', 'last_fullsized': '2024-04-25T01:23:27.835126+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T01:23:32.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-04-25T01:23:32.478 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.832949+0000', 'last_change': '2024-04-25T01:23:27.832949+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.832949+0000', 'last_undegraded': '2024-04-25T01:23:27.832949+0000', 'last_fullsized': '2024-04-25T01:23:27.832949+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.478 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-04-25T01:23:32.478 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.826687+0000', 'last_change': '2024-04-25T01:23:27.826687+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.826687+0000', 'last_undegraded': '2024-04-25T01:23:27.826687+0000', 'last_fullsized': '2024-04-25T01:23:27.826687+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T01:23:32.478 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-04-25T01:23:32.478 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.833838+0000', 'last_change': '2024-04-25T01:23:27.833838+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.833838+0000', 'last_undegraded': '2024-04-25T01:23:27.833838+0000', 'last_fullsized': '2024-04-25T01:23:27.833838+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T01:23:32.478 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-04-25T01:23:32.478 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.836435+0000', 'last_change': '2024-04-25T01:23:27.836435+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.836435+0000', 'last_undegraded': '2024-04-25T01:23:27.836435+0000', 'last_fullsized': '2024-04-25T01:23:27.836435+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T01:23:32.478 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.9 is not active+clean 2024-04-25T01:23:32.478 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-04-25T01:23:27.831341+0000', 'last_change': '2024-04-25T01:23:27.831341+0000', 'last_active': '2024-04-25T01:23:27.818947+0000', 'last_peered': '2024-04-25T01:23:27.818947+0000', 'last_clean': '2024-04-25T01:23:27.818947+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T01:23:27.831341+0000', 'last_undegraded': '2024-04-25T01:23:27.831341+0000', 'last_fullsized': '2024-04-25T01:23:27.831341+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'last_clean_scrub_stamp': '2024-04-25T01:23:27.818947+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T01:23:32.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-25T01:23:32.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.616+0000 7f905a7f9640 1 Processor -- start 2024-04-25T01:23:32.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.617+0000 7f905a7f9640 1 -- start start 2024-04-25T01:23:32.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.617+0000 7f905a7f9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f90540d4a10 0x7f90540cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.617+0000 7f905a7f9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90540d1e90 con 0x7f905415e4c0 2024-04-25T01:23:32.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.617+0000 7f905a7f9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90540b54a0 con 0x7f90540b3ee0 2024-04-25T01:23:32.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.617+0000 7f905a7f9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f90540d2130 con 0x7f90540d4a10 2024-04-25T01:23:32.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.617+0000 7f9053fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f905415e4c0 0x7f90540d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36886/0 (socket says 172.21.15.77:36886) 2024-04-25T01:23:32.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.617+0000 7f9053fff640 1 -- 172.21.15.77:0/3681418982 learned_addr learned my addr 172.21.15.77:0/3681418982 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:32.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2914151252 0 0) 0x7f90540d1e90 con 0x7f905415e4c0 2024-04-25T01:23:32.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f90537fe640 1 --2- 172.21.15.77:0/3681418982 >> v2:172.21.15.121:3300/0 conn(0x7f90540d4a10 0x7f90540cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f903c003520 con 0x7f905415e4c0 2024-04-25T01:23:32.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4224600445 0 0) 0x7f903c003520 con 0x7f905415e4c0 2024-04-25T01:23:32.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 >> v2:172.21.15.121:3300/0 conn(0x7f90540d4a10 msgr2=0x7f90540cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:32.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 --2- 172.21.15.77:0/3681418982 >> v2:172.21.15.121:3300/0 conn(0x7f90540d4a10 0x7f90540cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 >> v1:172.21.15.121:6789/0 conn(0x7f90540b3ee0 legacy=0x7f90540d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90540b5780 con 0x7f905415e4c0 2024-04-25T01:23:32.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f9048001670 con 0x7f905415e4c0 2024-04-25T01:23:32.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 >> v1:172.21.15.77:6789/0 conn(0x7f905415e4c0 legacy=0x7f90540d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 --2- 172.21.15.77:0/3681418982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f903c0039b0 0x7f903c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 --2- 172.21.15.77:0/3681418982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f903c004800 0x7f903c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 --2- 172.21.15.77:0/3681418982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 0x7f903c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f90540d2130 con 0x7f903c0051c0 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f903c00c160 con 0x7f903c004800 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f903c00c360 con 0x7f903c0039b0 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9058d96640 1 --2- 172.21.15.77:0/3681418982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 0x7f903c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9053fff640 1 --2- 172.21.15.77:0/3681418982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f903c0039b0 0x7f903c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f90537fe640 1 --2- 172.21.15.77:0/3681418982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f903c004800 0x7f903c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9058d96640 1 -- 172.21.15.77:0/3681418982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f903c0039b0 msgr2=0x7f903c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9058d96640 1 --2- 172.21.15.77:0/3681418982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f903c0039b0 0x7f903c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9058d96640 1 -- 172.21.15.77:0/3681418982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f903c004800 msgr2=0x7f903c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9058d96640 1 --2- 172.21.15.77:0/3681418982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f903c004800 0x7f903c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.618+0000 7f9058d96640 1 -- 172.21.15.77:0/3681418982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903c003520 con 0x7f903c0051c0 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.619+0000 7f9058d96640 1 -- 172.21.15.77:0/3681418982 reap_dead start 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.619+0000 7f9058d96640 1 --2- 172.21.15.77:0/3681418982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 0x7f903c00b9c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f904c001e50 tx=0x7f903c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8247dc8c6edb5b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.619+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c001a70 con 0x7f903c0051c0 2024-04-25T01:23:32.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.619+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f904c003970 con 0x7f903c0051c0 2024-04-25T01:23:32.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.619+0000 7f9052ffd640 1 -- 172.21.15.77:0/3681418982 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c003d60 con 0x7f903c0051c0 2024-04-25T01:23:32.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.620+0000 7f905a7f9640 1 -- 172.21.15.77:0/3681418982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 msgr2=0x7f903c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.620+0000 7f905a7f9640 1 --2- 172.21.15.77:0/3681418982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 0x7f903c00b9c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f904c001e50 tx=0x7f903c005830 comp rx=0 tx=0).stop 2024-04-25T01:23:32.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.620+0000 7f905a7f9640 1 -- 172.21.15.77:0/3681418982 shutdown_connections 2024-04-25T01:23:32.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.620+0000 7f905a7f9640 1 --2- 172.21.15.77:0/3681418982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 0x7f903c00b9c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.620+0000 7f905a7f9640 1 -- 172.21.15.77:0/3681418982 >> 172.21.15.77:0/3681418982 conn(0x7f90540d3ef0 msgr2=0x7f90540d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:32.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.620+0000 7f905a7f9640 1 -- 172.21.15.77:0/3681418982 shutdown_connections 2024-04-25T01:23:32.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.620+0000 7f905a7f9640 1 -- 172.21.15.77:0/3681418982 wait complete. 2024-04-25T01:23:32.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.621+0000 7f905a7f9640 1 Processor -- start 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.621+0000 7f905a7f9640 1 -- start start 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f905a7f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 0x7f905415f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f905a7f9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f905415bb80 0x7f90540c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f905a7f9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90540c7a20 0x7f90540c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f905a7f9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f905415ea20 con 0x7f903c0051c0 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f905a7f9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f905415e8a0 con 0x7f90540c7a20 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f905a7f9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f90540b4370 con 0x7f905415bb80 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9053fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 0x7f905415f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9053fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 0x7f905415f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42406/0 (socket says 172.21.15.77:42406) 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9053fff640 1 -- 172.21.15.77:0/4212841862 learned_addr learned my addr 172.21.15.77:0/4212841862 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9058d96640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90540c7a20 0x7f90540c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f90537fe640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f905415bb80 0x7f90540c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9053fff640 1 -- 172.21.15.77:0/4212841862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f905415bb80 msgr2=0x7f90540c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9053fff640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f905415bb80 0x7f90540c69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9053fff640 1 -- 172.21.15.77:0/4212841862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90540c7a20 msgr2=0x7f90540c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9053fff640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90540c7a20 0x7f90540c7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9053fff640 1 -- 172.21.15.77:0/4212841862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905415f850 con 0x7f903c0051c0 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9053fff640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 0x7f905415f270 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9048001050 tx=0x7f9048007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9050ff9640 1 -- 172.21.15.77:0/4212841862 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904800f960 con 0x7f903c0051c0 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.622+0000 7f9050ff9640 1 -- 172.21.15.77:0/4212841862 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9048018070 con 0x7f903c0051c0 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.623+0000 7f905a7f9640 1 -- 172.21.15.77:0/4212841862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905415fb30 con 0x7f903c0051c0 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.623+0000 7f9050ff9640 1 -- 172.21.15.77:0/4212841862 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904800fd80 con 0x7f903c0051c0 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.623+0000 7f905a7f9640 1 -- 172.21.15.77:0/4212841862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90541600f0 con 0x7f903c0051c0 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.624+0000 7f905a7f9640 1 -- 172.21.15.77:0/4212841862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9054068fb0 con 0x7f903c0051c0 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.624+0000 7f9050ff9640 1 -- 172.21.15.77:0/4212841862 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f904801d460 con 0x7f903c0051c0 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.624+0000 7f9050ff9640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f904006be30 0x7f904006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.625+0000 7f90537fe640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f904006be30 0x7f904006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.625+0000 7f9050ff9640 1 -- 172.21.15.77:0/4212841862 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f904808b1a0 con 0x7f903c0051c0 2024-04-25T01:23:32.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.625+0000 7f90537fe640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f904006be30 0x7f904006e2f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f905415f200 tx=0x7f9044008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.629+0000 7f9050ff9640 1 -- 172.21.15.77:0/4212841862 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9048059930 con 0x7f903c0051c0 2024-04-25T01:23:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.866+0000 7f905a7f9640 1 -- 172.21.15.77:0/4212841862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f9054073c20 con 0x7f903c0051c0 2024-04-25T01:23:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.866+0000 7f9050ff9640 1 -- 172.21.15.77:0/4212841862 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1453 (secure 0 0 0) 0x7f9048059230 con 0x7f903c0051c0 2024-04-25T01:23:32.848 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:32.848 INFO:teuthology.orchestra.run.smithi077.stdout:{"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":131,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":8,"num_up_osds":8,"osd_up_since":1714008092,"num_in_osds":8,"osd_in_since":1714008083,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":227115008,"bytes_avail":772866998272,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":27,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"b","status":"up:active","gid":5901}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-25T01:23:07.172492+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-25T01:23:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 -- 172.21.15.77:0/4212841862 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f904006be30 msgr2=0x7f904006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f904006be30 0x7f904006e2f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f905415f200 tx=0x7f9044008040 comp rx=0 tx=0).stop 2024-04-25T01:23:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 -- 172.21.15.77:0/4212841862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 msgr2=0x7f905415f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 0x7f905415f270 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9048001050 tx=0x7f9048007720 comp rx=0 tx=0).stop 2024-04-25T01:23:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 -- 172.21.15.77:0/4212841862 shutdown_connections 2024-04-25T01:23:32.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f904006be30 0x7f904006e2f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90540c7a20 0x7f90540c7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f905415bb80 0x7f90540c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 --2- 172.21.15.77:0/4212841862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f903c0051c0 0x7f905415f270 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 -- 172.21.15.77:0/4212841862 >> 172.21.15.77:0/4212841862 conn(0x7f90540d3ef0 msgr2=0x7f90540b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:32.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 -- 172.21.15.77:0/4212841862 shutdown_connections 2024-04-25T01:23:32.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.869+0000 7f905a7f9640 1 -- 172.21.15.77:0/4212841862 wait complete. 2024-04-25T01:23:32.861 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-25T01:23:32.861 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:23:32.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.976+0000 7f7fddc38640 1 Processor -- start 2024-04-25T01:23:32.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.976+0000 7f7fddc38640 1 -- start start 2024-04-25T01:23:32.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.977+0000 7f7fddc38640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fd80d4a10 0x7f7fd80cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.977+0000 7f7fddc38640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7fd80b3ee0 0x7f7fd80b4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.977+0000 7f7fddc38640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fd80d1e90 con 0x7f7fd815e4c0 2024-04-25T01:23:32.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.977+0000 7f7fddc38640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f7fd80b54a0 con 0x7f7fd80d4a10 2024-04-25T01:23:32.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.977+0000 7f7fddc38640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7fd80d2130 con 0x7f7fd80b3ee0 2024-04-25T01:23:32.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.977+0000 7f7fceffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fd80d4a10 0x7f7fd80cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.977+0000 7f7fceffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fd80d4a10 0x7f7fd80cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42418/0 (socket says 172.21.15.77:42418) 2024-04-25T01:23:32.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.977+0000 7f7fceffd640 1 -- 172.21.15.77:0/3724104631 learned_addr learned my addr 172.21.15.77:0/3724104631 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:32.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.978+0000 7f7fd7fff640 1 --2- 172.21.15.77:0/3724104631 >> v2:172.21.15.121:3300/0 conn(0x7f7fd80b3ee0 0x7f7fd80b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.978+0000 7f7fceffd640 1 -- 172.21.15.77:0/3724104631 >> v2:172.21.15.121:3300/0 conn(0x7f7fd80b3ee0 msgr2=0x7f7fd80b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:32.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.978+0000 7f7fceffd640 1 --2- 172.21.15.77:0/3724104631 >> v2:172.21.15.121:3300/0 conn(0x7f7fd80b3ee0 0x7f7fd80b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.978+0000 7f7fceffd640 1 -- 172.21.15.77:0/3724104631 >> v1:172.21.15.121:6789/0 conn(0x7f7fd815e4c0 legacy=0x7f7fd80d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:32.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.978+0000 7f7fceffd640 1 -- 172.21.15.77:0/3724104631 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd80b5780 con 0x7f7fd80d4a10 2024-04-25T01:23:32.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.978+0000 7f7fceffd640 1 --2- 172.21.15.77:0/3724104631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd80d4a10 0x7f7fd80cfaf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f7fd808d2e0 tx=0x7f7fc002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd27bab7020b73b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.979+0000 7f7fd6ffd640 1 -- 172.21.15.77:0/3724104631 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0037070 con 0x7f7fd80d4a10 2024-04-25T01:23:32.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.979+0000 7f7fd6ffd640 1 -- 172.21.15.77:0/3724104631 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fc0032440 con 0x7f7fd80d4a10 2024-04-25T01:23:32.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.979+0000 7f7fd6ffd640 1 -- 172.21.15.77:0/3724104631 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0032810 con 0x7f7fd80d4a10 2024-04-25T01:23:32.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.979+0000 7f7fddc38640 1 -- 172.21.15.77:0/3724104631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd80d4a10 msgr2=0x7f7fd80cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.979+0000 7f7fddc38640 1 --2- 172.21.15.77:0/3724104631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd80d4a10 0x7f7fd80cfaf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f7fd808d2e0 tx=0x7f7fc002a1a0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.979+0000 7f7fddc38640 1 -- 172.21.15.77:0/3724104631 shutdown_connections 2024-04-25T01:23:32.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.979+0000 7f7fddc38640 1 --2- 172.21.15.77:0/3724104631 >> v2:172.21.15.121:3300/0 conn(0x7f7fd80b3ee0 0x7f7fd80b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.979+0000 7f7fddc38640 1 --2- 172.21.15.77:0/3724104631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd80d4a10 0x7f7fd80cfaf0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.979+0000 7f7fddc38640 1 -- 172.21.15.77:0/3724104631 >> 172.21.15.77:0/3724104631 conn(0x7f7fd80d3ef0 msgr2=0x7f7fd80d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:32.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.979+0000 7f7fddc38640 1 -- 172.21.15.77:0/3724104631 shutdown_connections 2024-04-25T01:23:32.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.979+0000 7f7fddc38640 1 -- 172.21.15.77:0/3724104631 wait complete. 2024-04-25T01:23:32.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.980+0000 7f7fddc38640 1 Processor -- start 2024-04-25T01:23:32.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.980+0000 7f7fddc38640 1 -- start start 2024-04-25T01:23:32.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.981+0000 7f7fddc38640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd80b3ee0 0x7f7fd8161450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.981+0000 7f7fddc38640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd80d4a10 0x7f7fd80cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.981+0000 7f7fddc38640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd815e4c0 0x7f7fd80cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.981+0000 7f7fddc38640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd80b30a0 con 0x7f7fd815e4c0 2024-04-25T01:23:32.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.981+0000 7f7fddc38640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd80b2f20 con 0x7f7fd80b3ee0 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.981+0000 7f7fddc38640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7fd815e8a0 con 0x7f7fd80d4a10 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.981+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd815e4c0 0x7f7fd80cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.981+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd815e4c0 0x7f7fd80cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42422/0 (socket says 172.21.15.77:42422) 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.981+0000 7f7fd7fff640 1 -- 172.21.15.77:0/1218611182 learned_addr learned my addr 172.21.15.77:0/1218611182 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.981+0000 7f7fd77fe640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd80b3ee0 0x7f7fd8161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.981+0000 7f7fceffd640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd80d4a10 0x7f7fd80cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fd7fff640 1 -- 172.21.15.77:0/1218611182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd80d4a10 msgr2=0x7f7fd80cafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fd7fff640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd80d4a10 0x7f7fd80cafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fd7fff640 1 -- 172.21.15.77:0/1218611182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd80b3ee0 msgr2=0x7f7fd8161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fd7fff640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd80b3ee0 0x7f7fd8161450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fd7fff640 1 -- 172.21.15.77:0/1218611182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd8161990 con 0x7f7fd815e4c0 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fd77fe640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd80b3ee0 0x7f7fd8161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fd7fff640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd815e4c0 0x7f7fd80cc160 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0007e90 tx=0x7f7fd000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fd4ff9640 1 -- 172.21.15.77:0/1218611182 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0014420 con 0x7f7fd815e4c0 2024-04-25T01:23:32.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fd4ff9640 1 -- 172.21.15.77:0/1218611182 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fd0019070 con 0x7f7fd815e4c0 2024-04-25T01:23:32.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fddc38640 1 -- 172.21.15.77:0/1218611182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd80cba40 con 0x7f7fd815e4c0 2024-04-25T01:23:32.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fd4ff9640 1 -- 172.21.15.77:0/1218611182 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0014800 con 0x7f7fd815e4c0 2024-04-25T01:23:32.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.982+0000 7f7fddc38640 1 -- 172.21.15.77:0/1218611182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd81f5350 con 0x7f7fd815e4c0 2024-04-25T01:23:32.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.983+0000 7f7fddc38640 1 -- 172.21.15.77:0/1218611182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd8068fb0 con 0x7f7fd815e4c0 2024-04-25T01:23:32.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.983+0000 7f7fd4ff9640 1 -- 172.21.15.77:0/1218611182 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7fd00149a0 con 0x7f7fd815e4c0 2024-04-25T01:23:32.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.984+0000 7f7fd4ff9640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fb806be30 0x7f7fb806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.984+0000 7f7fd4ff9640 1 -- 172.21.15.77:0/1218611182 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f7fd008b890 con 0x7f7fd815e4c0 2024-04-25T01:23:32.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.984+0000 7f7fd77fe640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fb806be30 0x7f7fb806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.984+0000 7f7fd77fe640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fb806be30 0x7f7fb806e2f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7fd806d440 tx=0x7f7fc8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:32.988+0000 7f7fd4ff9640 1 -- 172.21.15.77:0/1218611182 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7fd005a020 con 0x7f7fd815e4c0 2024-04-25T01:23:33.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.158+0000 7f7fddc38640 1 -- 172.21.15.77:0/1218611182 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7fd80ce200 con 0x7f7fb806be30 2024-04-25T01:23:33.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.163+0000 7f7fd4ff9640 1 -- 172.21.15.77:0/1218611182 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246638 (secure 0 0 0) 0x7f7fd80ce200 con 0x7f7fb806be30 2024-04-25T01:23:33.147 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:33.149 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:23:33.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.166+0000 7f7fddc38640 1 -- 172.21.15.77:0/1218611182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fb806be30 msgr2=0x7f7fb806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:33.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.166+0000 7f7fddc38640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fb806be30 0x7f7fb806e2f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7fd806d440 tx=0x7f7fc8009040 comp rx=0 tx=0).stop 2024-04-25T01:23:33.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.166+0000 7f7fddc38640 1 -- 172.21.15.77:0/1218611182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd815e4c0 msgr2=0x7f7fd80cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:33.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.166+0000 7f7fddc38640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd815e4c0 0x7f7fd80cc160 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0007e90 tx=0x7f7fd000e4b0 comp rx=0 tx=0).stop 2024-04-25T01:23:33.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.166+0000 7f7fddc38640 1 -- 172.21.15.77:0/1218611182 shutdown_connections 2024-04-25T01:23:33.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.166+0000 7f7fddc38640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7fb806be30 0x7f7fb806e2f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:33.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.166+0000 7f7fddc38640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fd815e4c0 0x7f7fd80cc160 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:33.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.166+0000 7f7fddc38640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd80d4a10 0x7f7fd80cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:33.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.166+0000 7f7fddc38640 1 --2- 172.21.15.77:0/1218611182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd80b3ee0 0x7f7fd8161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:33.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.166+0000 7f7fddc38640 1 -- 172.21.15.77:0/1218611182 >> 172.21.15.77:0/1218611182 conn(0x7f7fd80d3ef0 msgr2=0x7f7fd80d2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:33.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.167+0000 7f7fddc38640 1 -- 172.21.15.77:0/1218611182 shutdown_connections 2024-04-25T01:23:33.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:33.167+0000 7f7fddc38640 1 -- 172.21.15.77:0/1218611182 wait complete. 2024-04-25T01:23:33.158 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-04-25T01:23:31.176800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221792,"kb_used_data":5608,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752928,"statfs":{"total":773094113280,"available":772866998272,"internally_reserved":0,"allocated":5742592,"data_stored":3179920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.354926"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828699+0000","last_change":"2024-04-25T01:23:27.828699+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828699+0000","last_undegraded":"2024-04-25T01:23:27.828699+0000","last_fullsized":"2024-04-25T01:23:27.828699+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833069+0000","last_change":"2024-04-25T01:23:27.833069+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833069+0000","last_undegraded":"2024-04-25T01:23:27.833069+0000","last_fullsized":"2024-04-25T01:23:27.833069+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838229+0000","last_change":"2024-04-25T01:23:27.838229+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838229+0000","last_undegraded":"2024-04-25T01:23:27.838229+0000","last_fullsized":"2024-04-25T01:23:27.838229+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830113+0000","last_change":"2024-04-25T01:23:27.830113+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830113+0000","last_undegraded":"2024-04-25T01:23:27.830113+0000","last_fullsized":"2024-04-25T01:23:27.830113+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830687+0000","last_change":"2024-04-25T01:23:27.830687+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830687+0000","last_undegraded":"2024-04-25T01:23:27.830687+0000","last_fullsized":"2024-04-25T01:23:27.830687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828146+0000","last_change":"2024-04-25T01:23:27.828146+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828146+0000","last_undegraded":"2024-04-25T01:23:27.828146+0000","last_fullsized":"2024-04-25T01:23:27.828146+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838894+0000","last_change":"2024-04-25T01:23:27.838894+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838894+0000","last_undegraded":"2024-04-25T01:23:27.838894+0000","last_fullsized":"2024-04-25T01:23:27.838894+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834175+0000","last_change":"2024-04-25T01:23:27.834175+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834175+0000","last_undegraded":"2024-04-25T01:23:27.834175+0000","last_fullsized":"2024-04-25T01:23:27.834175+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836796+0000","last_change":"2024-04-25T01:23:27.836796+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836796+0000","last_undegraded":"2024-04-25T01:23:27.836796+0000","last_fullsized":"2024-04-25T01:23:27.836796+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827603+0000","last_change":"2024-04-25T01:23:27.827603+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827603+0000","last_undegraded":"2024-04-25T01:23:27.827603+0000","last_fullsized":"2024-04-25T01:23:27.827603+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830254+0000","last_change":"2024-04-25T01:23:27.830254+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830254+0000","last_undegraded":"2024-04-25T01:23:27.830254+0000","last_fullsized":"2024-04-25T01:23:27.830254+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833652+0000","last_change":"2024-04-25T01:23:27.833652+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833652+0000","last_undegraded":"2024-04-25T01:23:27.833652+0000","last_fullsized":"2024-04-25T01:23:27.833652+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827511+0000","last_change":"2024-04-25T01:23:27.827511+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827511+0000","last_undegraded":"2024-04-25T01:23:27.827511+0000","last_fullsized":"2024-04-25T01:23:27.827511+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838536+0000","last_change":"2024-04-25T01:23:27.838536+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838536+0000","last_undegraded":"2024-04-25T01:23:27.838536+0000","last_fullsized":"2024-04-25T01:23:27.838536+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.829534+0000","last_change":"2024-04-25T01:23:27.829534+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.829534+0000","last_undegraded":"2024-04-25T01:23:27.829534+0000","last_fullsized":"2024-04-25T01:23:27.829534+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.825966+0000","last_change":"2024-04-25T01:23:27.825966+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.825966+0000","last_undegraded":"2024-04-25T01:23:27.825966+0000","last_fullsized":"2024-04-25T01:23:27.825966+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.838142+0000","last_change":"2024-04-25T01:23:27.838142+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.838142+0000","last_undegraded":"2024-04-25T01:23:27.838142+0000","last_fullsized":"2024-04-25T01:23:27.838142+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832007+0000","last_change":"2024-04-25T01:23:27.832007+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832007+0000","last_undegraded":"2024-04-25T01:23:27.832007+0000","last_fullsized":"2024-04-25T01:23:27.832007+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836983+0000","last_change":"2024-04-25T01:23:27.836983+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836983+0000","last_undegraded":"2024-04-25T01:23:27.836983+0000","last_fullsized":"2024-04-25T01:23:27.836983+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832039+0000","last_change":"2024-04-25T01:23:27.832039+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832039+0000","last_undegraded":"2024-04-25T01:23:27.832039+0000","last_fullsized":"2024-04-25T01:23:27.832039+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830258+0000","last_change":"2024-04-25T01:23:27.830258+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830258+0000","last_undegraded":"2024-04-25T01:23:27.830258+0000","last_fullsized":"2024-04-25T01:23:27.830258+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833623+0000","last_change":"2024-04-25T01:23:27.833623+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833623+0000","last_undegraded":"2024-04-25T01:23:27.833623+0000","last_fullsized":"2024-04-25T01:23:27.833623+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830382+0000","last_change":"2024-04-25T01:23:27.830382+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830382+0000","last_undegraded":"2024-04-25T01:23:27.830382+0000","last_fullsized":"2024-04-25T01:23:27.830382+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836595+0000","last_change":"2024-04-25T01:23:27.836595+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836595+0000","last_undegraded":"2024-04-25T01:23:27.836595+0000","last_fullsized":"2024-04-25T01:23:27.836595+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836402+0000","last_change":"2024-04-25T01:23:27.836402+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836402+0000","last_undegraded":"2024-04-25T01:23:27.836402+0000","last_fullsized":"2024-04-25T01:23:27.836402+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835991+0000","last_change":"2024-04-25T01:23:27.835991+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835991+0000","last_undegraded":"2024-04-25T01:23:27.835991+0000","last_fullsized":"2024-04-25T01:23:27.835991+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827570+0000","last_change":"2024-04-25T01:23:27.827570+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827570+0000","last_undegraded":"2024-04-25T01:23:27.827570+0000","last_fullsized":"2024-04-25T01:23:27.827570+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831597+0000","last_change":"2024-04-25T01:23:27.831597+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831597+0000","last_undegraded":"2024-04-25T01:23:27.831597+0000","last_fullsized":"2024-04-25T01:23:27.831597+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836573+0000","last_change":"2024-04-25T01:23:27.836573+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836573+0000","last_undegraded":"2024-04-25T01:23:27.836573+0000","last_fullsized":"2024-04-25T01:23:27.836573+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830474+0000","last_change":"2024-04-25T01:23:27.830474+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830474+0000","last_undegraded":"2024-04-25T01:23:27.830474+0000","last_fullsized":"2024-04-25T01:23:27.830474+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835732+0000","last_change":"2024-04-25T01:23:27.835732+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835732+0000","last_undegraded":"2024-04-25T01:23:27.835732+0000","last_fullsized":"2024-04-25T01:23:27.835732+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.828172+0000","last_change":"2024-04-25T01:23:27.828172+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.828172+0000","last_undegraded":"2024-04-25T01:23:27.828172+0000","last_fullsized":"2024-04-25T01:23:27.828172+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830914+0000","last_change":"2024-04-25T01:23:27.830914+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830914+0000","last_undegraded":"2024-04-25T01:23:27.830914+0000","last_fullsized":"2024-04-25T01:23:27.830914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844780+0000","last_change":"2024-04-25T01:23:27.840369+0000","last_active":"2024-04-25T01:23:27.844780+0000","last_peered":"2024-04-25T01:23:27.844780+0000","last_clean":"2024-04-25T01:23:27.844780+0000","last_became_active":"2024-04-25T01:23:27.839987+0000","last_became_peered":"2024-04-25T01:23:27.839987+0000","last_unstale":"2024-04-25T01:23:27.844780+0000","last_undegraded":"2024-04-25T01:23:27.844780+0000","last_fullsized":"2024-04-25T01:23:27.844780+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.219905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826648+0000","last_change":"2024-04-25T01:23:27.826648+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826648+0000","last_undegraded":"2024-04-25T01:23:27.826648+0000","last_fullsized":"2024-04-25T01:23:27.826648+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.841990+0000","last_change":"2024-04-25T01:23:27.838394+0000","last_active":"2024-04-25T01:23:27.841990+0000","last_peered":"2024-04-25T01:23:27.841990+0000","last_clean":"2024-04-25T01:23:27.841990+0000","last_became_active":"2024-04-25T01:23:27.838114+0000","last_became_peered":"2024-04-25T01:23:27.838114+0000","last_unstale":"2024-04-25T01:23:27.841990+0000","last_undegraded":"2024-04-25T01:23:27.841990+0000","last_fullsized":"2024-04-25T01:23:27.841990+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:48.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832822+0000","last_change":"2024-04-25T01:23:27.832822+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832822+0000","last_undegraded":"2024-04-25T01:23:27.832822+0000","last_fullsized":"2024-04-25T01:23:27.832822+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.842100+0000","last_change":"2024-04-25T01:23:27.839860+0000","last_active":"2024-04-25T01:23:27.842100+0000","last_peered":"2024-04-25T01:23:27.842100+0000","last_clean":"2024-04-25T01:23:27.842100+0000","last_became_active":"2024-04-25T01:23:27.839589+0000","last_became_peered":"2024-04-25T01:23:27.839589+0000","last_unstale":"2024-04-25T01:23:27.842100+0000","last_undegraded":"2024-04-25T01:23:27.842100+0000","last_fullsized":"2024-04-25T01:23:27.842100+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:52.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835561+0000","last_change":"2024-04-25T01:23:27.835561+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835561+0000","last_undegraded":"2024-04-25T01:23:27.835561+0000","last_fullsized":"2024-04-25T01:23:27.835561+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847649+0000","last_change":"2024-04-25T01:23:27.842142+0000","last_active":"2024-04-25T01:23:27.847649+0000","last_peered":"2024-04-25T01:23:27.847649+0000","last_clean":"2024-04-25T01:23:27.847649+0000","last_became_active":"2024-04-25T01:23:27.841740+0000","last_became_peered":"2024-04-25T01:23:27.841740+0000","last_unstale":"2024-04-25T01:23:27.847649+0000","last_undegraded":"2024-04-25T01:23:27.847649+0000","last_fullsized":"2024-04-25T01:23:27.847649+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:13.292186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831025+0000","last_change":"2024-04-25T01:23:27.831025+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831025+0000","last_undegraded":"2024-04-25T01:23:27.831025+0000","last_fullsized":"2024-04-25T01:23:27.831025+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844533+0000","last_change":"2024-04-25T01:23:27.831214+0000","last_active":"2024-04-25T01:23:27.844533+0000","last_peered":"2024-04-25T01:23:27.844533+0000","last_clean":"2024-04-25T01:23:27.844533+0000","last_became_active":"2024-04-25T01:23:27.830883+0000","last_became_peered":"2024-04-25T01:23:27.830883+0000","last_unstale":"2024-04-25T01:23:27.844533+0000","last_undegraded":"2024-04-25T01:23:27.844533+0000","last_fullsized":"2024-04-25T01:23:27.844533+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:40.061850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830940+0000","last_change":"2024-04-25T01:23:27.830940+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830940+0000","last_undegraded":"2024-04-25T01:23:27.830940+0000","last_fullsized":"2024-04-25T01:23:27.830940+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844655+0000","last_change":"2024-04-25T01:23:27.839506+0000","last_active":"2024-04-25T01:23:27.844655+0000","last_peered":"2024-04-25T01:23:27.844655+0000","last_clean":"2024-04-25T01:23:27.844655+0000","last_became_active":"2024-04-25T01:23:27.838600+0000","last_became_peered":"2024-04-25T01:23:27.838600+0000","last_unstale":"2024-04-25T01:23:27.844655+0000","last_undegraded":"2024-04-25T01:23:27.844655+0000","last_fullsized":"2024-04-25T01:23:27.844655+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:48.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833846+0000","last_change":"2024-04-25T01:23:27.833846+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833846+0000","last_undegraded":"2024-04-25T01:23:27.833846+0000","last_fullsized":"2024-04-25T01:23:27.833846+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847468+0000","last_change":"2024-04-25T01:23:27.843276+0000","last_active":"2024-04-25T01:23:27.847468+0000","last_peered":"2024-04-25T01:23:27.847468+0000","last_clean":"2024-04-25T01:23:27.847468+0000","last_became_active":"2024-04-25T01:23:27.842824+0000","last_became_peered":"2024-04-25T01:23:27.842824+0000","last_unstale":"2024-04-25T01:23:27.847468+0000","last_undegraded":"2024-04-25T01:23:27.847468+0000","last_fullsized":"2024-04-25T01:23:27.847468+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:48.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831669+0000","last_change":"2024-04-25T01:23:27.831669+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831669+0000","last_undegraded":"2024-04-25T01:23:27.831669+0000","last_fullsized":"2024-04-25T01:23:27.831669+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848229+0000","last_change":"2024-04-25T01:23:27.842530+0000","last_active":"2024-04-25T01:23:27.848229+0000","last_peered":"2024-04-25T01:23:27.848229+0000","last_clean":"2024-04-25T01:23:27.848229+0000","last_became_active":"2024-04-25T01:23:27.842265+0000","last_became_peered":"2024-04-25T01:23:27.842265+0000","last_unstale":"2024-04-25T01:23:27.848229+0000","last_undegraded":"2024-04-25T01:23:27.848229+0000","last_fullsized":"2024-04-25T01:23:27.848229+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:49:28.578991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.829914+0000","last_change":"2024-04-25T01:23:27.829914+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.829914+0000","last_undegraded":"2024-04-25T01:23:27.829914+0000","last_fullsized":"2024-04-25T01:23:27.829914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847439+0000","last_change":"2024-04-25T01:23:27.842714+0000","last_active":"2024-04-25T01:23:27.847439+0000","last_peered":"2024-04-25T01:23:27.847439+0000","last_clean":"2024-04-25T01:23:27.847439+0000","last_became_active":"2024-04-25T01:23:27.842268+0000","last_became_peered":"2024-04-25T01:23:27.842268+0000","last_unstale":"2024-04-25T01:23:27.847439+0000","last_undegraded":"2024-04-25T01:23:27.847439+0000","last_fullsized":"2024-04-25T01:23:27.847439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:01.477667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827321+0000","last_change":"2024-04-25T01:23:27.827321+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827321+0000","last_undegraded":"2024-04-25T01:23:27.827321+0000","last_fullsized":"2024-04-25T01:23:27.827321+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844619+0000","last_change":"2024-04-25T01:23:27.839254+0000","last_active":"2024-04-25T01:23:27.844619+0000","last_peered":"2024-04-25T01:23:27.844619+0000","last_clean":"2024-04-25T01:23:27.844619+0000","last_became_active":"2024-04-25T01:23:27.838972+0000","last_became_peered":"2024-04-25T01:23:27.838972+0000","last_unstale":"2024-04-25T01:23:27.844619+0000","last_undegraded":"2024-04-25T01:23:27.844619+0000","last_fullsized":"2024-04-25T01:23:27.844619+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:04.719084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834497+0000","last_change":"2024-04-25T01:23:27.834497+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834497+0000","last_undegraded":"2024-04-25T01:23:27.834497+0000","last_fullsized":"2024-04-25T01:23:27.834497+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847498+0000","last_change":"2024-04-25T01:23:27.843358+0000","last_active":"2024-04-25T01:23:27.847498+0000","last_peered":"2024-04-25T01:23:27.847498+0000","last_clean":"2024-04-25T01:23:27.847498+0000","last_became_active":"2024-04-25T01:23:27.843098+0000","last_became_peered":"2024-04-25T01:23:27.843098+0000","last_unstale":"2024-04-25T01:23:27.847498+0000","last_undegraded":"2024-04-25T01:23:27.847498+0000","last_fullsized":"2024-04-25T01:23:27.847498+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:38.211801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830273+0000","last_change":"2024-04-25T01:23:27.830273+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830273+0000","last_undegraded":"2024-04-25T01:23:27.830273+0000","last_fullsized":"2024-04-25T01:23:27.830273+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846301+0000","last_change":"2024-04-25T01:23:27.841668+0000","last_active":"2024-04-25T01:23:27.846301+0000","last_peered":"2024-04-25T01:23:27.846301+0000","last_clean":"2024-04-25T01:23:27.846301+0000","last_became_active":"2024-04-25T01:23:27.841382+0000","last_became_peered":"2024-04-25T01:23:27.841382+0000","last_unstale":"2024-04-25T01:23:27.846301+0000","last_undegraded":"2024-04-25T01:23:27.846301+0000","last_fullsized":"2024-04-25T01:23:27.846301+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:00:11.736192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827805+0000","last_change":"2024-04-25T01:23:27.827805+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827805+0000","last_undegraded":"2024-04-25T01:23:27.827805+0000","last_fullsized":"2024-04-25T01:23:27.827805+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848483+0000","last_change":"2024-04-25T01:23:27.845237+0000","last_active":"2024-04-25T01:23:27.848483+0000","last_peered":"2024-04-25T01:23:27.848483+0000","last_clean":"2024-04-25T01:23:27.848483+0000","last_became_active":"2024-04-25T01:23:27.844336+0000","last_became_peered":"2024-04-25T01:23:27.844336+0000","last_unstale":"2024-04-25T01:23:27.848483+0000","last_undegraded":"2024-04-25T01:23:27.848483+0000","last_fullsized":"2024-04-25T01:23:27.848483+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:52.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834532+0000","last_change":"2024-04-25T01:23:27.834532+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834532+0000","last_undegraded":"2024-04-25T01:23:27.834532+0000","last_fullsized":"2024-04-25T01:23:27.834532+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.849509+0000","last_change":"2024-04-25T01:23:27.848962+0000","last_active":"2024-04-25T01:23:27.849509+0000","last_peered":"2024-04-25T01:23:27.849509+0000","last_clean":"2024-04-25T01:23:27.849509+0000","last_became_active":"2024-04-25T01:23:27.848694+0000","last_became_peered":"2024-04-25T01:23:27.848694+0000","last_unstale":"2024-04-25T01:23:27.849509+0000","last_undegraded":"2024-04-25T01:23:27.849509+0000","last_fullsized":"2024-04-25T01:23:27.849509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:04.282731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.834242+0000","last_change":"2024-04-25T01:23:27.834242+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.834242+0000","last_undegraded":"2024-04-25T01:23:27.834242+0000","last_fullsized":"2024-04-25T01:23:27.834242+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847532+0000","last_change":"2024-04-25T01:23:27.843254+0000","last_active":"2024-04-25T01:23:27.847532+0000","last_peered":"2024-04-25T01:23:27.847532+0000","last_clean":"2024-04-25T01:23:27.847532+0000","last_became_active":"2024-04-25T01:23:27.842506+0000","last_became_peered":"2024-04-25T01:23:27.842506+0000","last_unstale":"2024-04-25T01:23:27.847532+0000","last_undegraded":"2024-04-25T01:23:27.847532+0000","last_fullsized":"2024-04-25T01:23:27.847532+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:42.248852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835026+0000","last_change":"2024-04-25T01:23:27.835026+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835026+0000","last_undegraded":"2024-04-25T01:23:27.835026+0000","last_fullsized":"2024-04-25T01:23:27.835026+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846277+0000","last_change":"2024-04-25T01:23:27.841058+0000","last_active":"2024-04-25T01:23:27.846277+0000","last_peered":"2024-04-25T01:23:27.846277+0000","last_clean":"2024-04-25T01:23:27.846277+0000","last_became_active":"2024-04-25T01:23:27.840808+0000","last_became_peered":"2024-04-25T01:23:27.840808+0000","last_unstale":"2024-04-25T01:23:27.846277+0000","last_undegraded":"2024-04-25T01:23:27.846277+0000","last_fullsized":"2024-04-25T01:23:27.846277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:03.437919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833035+0000","last_change":"2024-04-25T01:23:27.833035+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833035+0000","last_undegraded":"2024-04-25T01:23:27.833035+0000","last_fullsized":"2024-04-25T01:23:27.833035+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848509+0000","last_change":"2024-04-25T01:23:27.846019+0000","last_active":"2024-04-25T01:23:27.848509+0000","last_peered":"2024-04-25T01:23:27.848509+0000","last_clean":"2024-04-25T01:23:27.848509+0000","last_became_active":"2024-04-25T01:23:27.845056+0000","last_became_peered":"2024-04-25T01:23:27.845056+0000","last_unstale":"2024-04-25T01:23:27.848509+0000","last_undegraded":"2024-04-25T01:23:27.848509+0000","last_fullsized":"2024-04-25T01:23:27.848509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:08.739387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826903+0000","last_change":"2024-04-25T01:23:27.826903+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826903+0000","last_undegraded":"2024-04-25T01:23:27.826903+0000","last_fullsized":"2024-04-25T01:23:27.826903+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.845756+0000","last_change":"2024-04-25T01:23:27.839989+0000","last_active":"2024-04-25T01:23:27.845756+0000","last_peered":"2024-04-25T01:23:27.845756+0000","last_clean":"2024-04-25T01:23:27.845756+0000","last_became_active":"2024-04-25T01:23:27.839737+0000","last_became_peered":"2024-04-25T01:23:27.839737+0000","last_unstale":"2024-04-25T01:23:27.845756+0000","last_undegraded":"2024-04-25T01:23:27.845756+0000","last_fullsized":"2024-04-25T01:23:27.845756+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:17.100820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.830591+0000","last_change":"2024-04-25T01:23:27.830591+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.830591+0000","last_undegraded":"2024-04-25T01:23:27.830591+0000","last_fullsized":"2024-04-25T01:23:27.830591+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.845062+0000","last_change":"2024-04-25T01:23:27.838563+0000","last_active":"2024-04-25T01:23:27.845062+0000","last_peered":"2024-04-25T01:23:27.845062+0000","last_clean":"2024-04-25T01:23:27.845062+0000","last_became_active":"2024-04-25T01:23:27.837764+0000","last_became_peered":"2024-04-25T01:23:27.837764+0000","last_unstale":"2024-04-25T01:23:27.845062+0000","last_undegraded":"2024-04-25T01:23:27.845062+0000","last_fullsized":"2024-04-25T01:23:27.845062+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:26.874008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826397+0000","last_change":"2024-04-25T01:23:27.826397+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826397+0000","last_undegraded":"2024-04-25T01:23:27.826397+0000","last_fullsized":"2024-04-25T01:23:27.826397+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847457+0000","last_change":"2024-04-25T01:23:27.843101+0000","last_active":"2024-04-25T01:23:27.847457+0000","last_peered":"2024-04-25T01:23:27.847457+0000","last_clean":"2024-04-25T01:23:27.847457+0000","last_became_active":"2024-04-25T01:23:27.842810+0000","last_became_peered":"2024-04-25T01:23:27.842810+0000","last_unstale":"2024-04-25T01:23:27.847457+0000","last_undegraded":"2024-04-25T01:23:27.847457+0000","last_fullsized":"2024-04-25T01:23:27.847457+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:04.719084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826783+0000","last_change":"2024-04-25T01:23:27.826783+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826783+0000","last_undegraded":"2024-04-25T01:23:27.826783+0000","last_fullsized":"2024-04-25T01:23:27.826783+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.838395+0000","last_change":"2024-04-25T01:23:27.834285+0000","last_active":"2024-04-25T01:23:27.838395+0000","last_peered":"2024-04-25T01:23:27.838395+0000","last_clean":"2024-04-25T01:23:27.838395+0000","last_became_active":"2024-04-25T01:23:27.834030+0000","last_became_peered":"2024-04-25T01:23:27.834030+0000","last_unstale":"2024-04-25T01:23:27.838395+0000","last_undegraded":"2024-04-25T01:23:27.838395+0000","last_fullsized":"2024-04-25T01:23:27.838395+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:33.027603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832642+0000","last_change":"2024-04-25T01:23:27.832642+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832642+0000","last_undegraded":"2024-04-25T01:23:27.832642+0000","last_fullsized":"2024-04-25T01:23:27.832642+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846225+0000","last_change":"2024-04-25T01:23:27.839744+0000","last_active":"2024-04-25T01:23:27.846225+0000","last_peered":"2024-04-25T01:23:27.846225+0000","last_clean":"2024-04-25T01:23:27.846225+0000","last_became_active":"2024-04-25T01:23:27.839478+0000","last_became_peered":"2024-04-25T01:23:27.839478+0000","last_unstale":"2024-04-25T01:23:27.846225+0000","last_undegraded":"2024-04-25T01:23:27.846225+0000","last_fullsized":"2024-04-25T01:23:27.846225+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:44.264978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833162+0000","last_change":"2024-04-25T01:23:27.833162+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833162+0000","last_undegraded":"2024-04-25T01:23:27.833162+0000","last_fullsized":"2024-04-25T01:23:27.833162+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848293+0000","last_change":"2024-04-25T01:23:27.843276+0000","last_active":"2024-04-25T01:23:27.848293+0000","last_peered":"2024-04-25T01:23:27.848293+0000","last_clean":"2024-04-25T01:23:27.848293+0000","last_became_active":"2024-04-25T01:23:27.842913+0000","last_became_peered":"2024-04-25T01:23:27.842913+0000","last_unstale":"2024-04-25T01:23:27.848293+0000","last_undegraded":"2024-04-25T01:23:27.848293+0000","last_fullsized":"2024-04-25T01:23:27.848293+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:33.027603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826481+0000","last_change":"2024-04-25T01:23:27.826481+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826481+0000","last_undegraded":"2024-04-25T01:23:27.826481+0000","last_fullsized":"2024-04-25T01:23:27.826481+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846474+0000","last_change":"2024-04-25T01:23:27.843723+0000","last_active":"2024-04-25T01:23:27.846474+0000","last_peered":"2024-04-25T01:23:27.846474+0000","last_clean":"2024-04-25T01:23:27.846474+0000","last_became_active":"2024-04-25T01:23:27.843464+0000","last_became_peered":"2024-04-25T01:23:27.843464+0000","last_unstale":"2024-04-25T01:23:27.846474+0000","last_undegraded":"2024-04-25T01:23:27.846474+0000","last_fullsized":"2024-04-25T01:23:27.846474+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:01.926860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826655+0000","last_change":"2024-04-25T01:23:27.826655+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826655+0000","last_undegraded":"2024-04-25T01:23:27.826655+0000","last_fullsized":"2024-04-25T01:23:27.826655+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848098+0000","last_change":"2024-04-25T01:23:27.847328+0000","last_active":"2024-04-25T01:23:27.848098+0000","last_peered":"2024-04-25T01:23:27.848098+0000","last_clean":"2024-04-25T01:23:27.848098+0000","last_became_active":"2024-04-25T01:23:27.847046+0000","last_became_peered":"2024-04-25T01:23:27.847046+0000","last_unstale":"2024-04-25T01:23:27.848098+0000","last_undegraded":"2024-04-25T01:23:27.848098+0000","last_fullsized":"2024-04-25T01:23:27.848098+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:59.145724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.827287+0000","last_change":"2024-04-25T01:23:27.827287+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.827287+0000","last_undegraded":"2024-04-25T01:23:27.827287+0000","last_fullsized":"2024-04-25T01:23:27.827287+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.847476+0000","last_change":"2024-04-25T01:23:27.842809+0000","last_active":"2024-04-25T01:23:27.847476+0000","last_peered":"2024-04-25T01:23:27.847476+0000","last_clean":"2024-04-25T01:23:27.847476+0000","last_became_active":"2024-04-25T01:23:27.842561+0000","last_became_peered":"2024-04-25T01:23:27.842561+0000","last_unstale":"2024-04-25T01:23:27.847476+0000","last_undegraded":"2024-04-25T01:23:27.847476+0000","last_fullsized":"2024-04-25T01:23:27.847476+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:28:34.927620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.835126+0000","last_change":"2024-04-25T01:23:27.835126+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.835126+0000","last_undegraded":"2024-04-25T01:23:27.835126+0000","last_fullsized":"2024-04-25T01:23:27.835126+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.848367+0000","last_change":"2024-04-25T01:23:27.844808+0000","last_active":"2024-04-25T01:23:27.848367+0000","last_peered":"2024-04-25T01:23:27.848367+0000","last_clean":"2024-04-25T01:23:27.848367+0000","last_became_active":"2024-04-25T01:23:27.844541+0000","last_became_peered":"2024-04-25T01:23:27.844541+0000","last_unstale":"2024-04-25T01:23:27.848367+0000","last_undegraded":"2024-04-25T01:23:27.848367+0000","last_fullsized":"2024-04-25T01:23:27.848367+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:48.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.832949+0000","last_change":"2024-04-25T01:23:27.832949+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.832949+0000","last_undegraded":"2024-04-25T01:23:27.832949+0000","last_fullsized":"2024-04-25T01:23:27.832949+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846394+0000","last_change":"2024-04-25T01:23:27.842454+0000","last_active":"2024-04-25T01:23:27.846394+0000","last_peered":"2024-04-25T01:23:27.846394+0000","last_clean":"2024-04-25T01:23:27.846394+0000","last_became_active":"2024-04-25T01:23:27.842209+0000","last_became_peered":"2024-04-25T01:23:27.842209+0000","last_unstale":"2024-04-25T01:23:27.846394+0000","last_undegraded":"2024-04-25T01:23:27.846394+0000","last_fullsized":"2024-04-25T01:23:27.846394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:22.705548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.826687+0000","last_change":"2024-04-25T01:23:27.826687+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.826687+0000","last_undegraded":"2024-04-25T01:23:27.826687+0000","last_fullsized":"2024-04-25T01:23:27.826687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846248+0000","last_change":"2024-04-25T01:23:27.841080+0000","last_active":"2024-04-25T01:23:27.846248+0000","last_peered":"2024-04-25T01:23:27.846248+0000","last_clean":"2024-04-25T01:23:27.846248+0000","last_became_active":"2024-04-25T01:23:27.840813+0000","last_became_peered":"2024-04-25T01:23:27.840813+0000","last_unstale":"2024-04-25T01:23:27.846248+0000","last_undegraded":"2024-04-25T01:23:27.846248+0000","last_fullsized":"2024-04-25T01:23:27.846248+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:44.720263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.833838+0000","last_change":"2024-04-25T01:23:27.833838+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.833838+0000","last_undegraded":"2024-04-25T01:23:27.833838+0000","last_fullsized":"2024-04-25T01:23:27.833838+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.844601+0000","last_change":"2024-04-25T01:23:27.841167+0000","last_active":"2024-04-25T01:23:27.844601+0000","last_peered":"2024-04-25T01:23:27.844601+0000","last_clean":"2024-04-25T01:23:27.844601+0000","last_became_active":"2024-04-25T01:23:27.840932+0000","last_became_peered":"2024-04-25T01:23:27.840932+0000","last_unstale":"2024-04-25T01:23:27.844601+0000","last_undegraded":"2024-04-25T01:23:27.844601+0000","last_fullsized":"2024-04-25T01:23:27.844601+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:59.145724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.836435+0000","last_change":"2024-04-25T01:23:27.836435+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.836435+0000","last_undegraded":"2024-04-25T01:23:27.836435+0000","last_fullsized":"2024-04-25T01:23:27.836435+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.842097+0000","last_change":"2024-04-25T01:23:27.838507+0000","last_active":"2024-04-25T01:23:27.842097+0000","last_peered":"2024-04-25T01:23:27.842097+0000","last_clean":"2024-04-25T01:23:27.842097+0000","last_became_active":"2024-04-25T01:23:27.838016+0000","last_became_peered":"2024-04-25T01:23:27.838016+0000","last_unstale":"2024-04-25T01:23:27.842097+0000","last_undegraded":"2024-04-25T01:23:27.842097+0000","last_fullsized":"2024-04-25T01:23:27.842097+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:09.155513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-04-25T01:23:27.831341+0000","last_change":"2024-04-25T01:23:27.831341+0000","last_active":"2024-04-25T01:23:27.818947+0000","last_peered":"2024-04-25T01:23:27.818947+0000","last_clean":"2024-04-25T01:23:27.818947+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T01:23:27.831341+0000","last_undegraded":"2024-04-25T01:23:27.831341+0000","last_fullsized":"2024-04-25T01:23:27.831341+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.846638+0000","last_change":"2024-04-25T01:23:27.841576+0000","last_active":"2024-04-25T01:23:27.846638+0000","last_peered":"2024-04-25T01:23:27.846638+0000","last_clean":"2024-04-25T01:23:27.846638+0000","last_became_active":"2024-04-25T01:23:27.839038+0000","last_became_peered":"2024-04-25T01:23:27.839038+0000","last_unstale":"2024-04-25T01:23:27.846638+0000","last_undegraded":"2024-04-25T01:23:27.846638+0000","last_fullsized":"2024-04-25T01:23:27.846638+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:26:54.800425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":222,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:23:27.833349+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:23:27.833349+0000","last_peered":"2024-04-25T01:23:27.833349+0000","last_clean":"2024-04-25T01:23:27.833349+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:23:27.833349+0000","last_undegraded":"2024-04-25T01:23:27.833349+0000","last_fullsized":"2024-04-25T01:23:27.833349+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1163264,"data_stored":840266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":249898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1163264,"data_stored":840266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:23:34.087 INFO:tasks.ceph.mds.b.smithi121.stderr:2024-04-25T01:23:34.107+0000 7feabb8b1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:23:36.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:23:36.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.291+0000 7f8da8488640 1 Processor -- start 2024-04-25T01:23:36.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.291+0000 7f8da8488640 1 -- start start 2024-04-25T01:23:36.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.292+0000 7f8da8488640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8da015e4c0 0x7f8da00d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.292+0000 7f8da8488640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8da00d2d40 0x7f8da00cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.292+0000 7f8da8488640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8da00d1e90 con 0x7f8da00d4a10 2024-04-25T01:23:36.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.292+0000 7f8da8488640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f8da00b54a0 con 0x7f8da015e4c0 2024-04-25T01:23:36.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.292+0000 7f8da8488640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8da00d2130 con 0x7f8da00d2d40 2024-04-25T01:23:36.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.293+0000 7f8da6224640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8da015e4c0 0x7f8da00d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.293+0000 7f8da6224640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8da015e4c0 0x7f8da00d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42432/0 (socket says 172.21.15.77:42432) 2024-04-25T01:23:36.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.293+0000 7f8da6224640 1 -- 172.21.15.77:0/2502737946 learned_addr learned my addr 172.21.15.77:0/2502737946 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:36.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.293+0000 7f8da5a23640 1 --1- 172.21.15.77:0/2502737946 >> v1:172.21.15.77:6789/0 conn(0x7f8da00d4a10 0x7f8da00b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36902/0 (socket says 172.21.15.77:36902) 2024-04-25T01:23:36.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.293+0000 7f8da6a25640 1 --2- 172.21.15.77:0/2502737946 >> v2:172.21.15.121:3300/0 conn(0x7f8da00d2d40 0x7f8da00cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.293+0000 7f8da6224640 1 -- 172.21.15.77:0/2502737946 >> v2:172.21.15.121:3300/0 conn(0x7f8da00d2d40 msgr2=0x7f8da00cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:36.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.293+0000 7f8da6224640 1 --2- 172.21.15.77:0/2502737946 >> v2:172.21.15.121:3300/0 conn(0x7f8da00d2d40 0x7f8da00cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.293+0000 7f8da6224640 1 -- 172.21.15.77:0/2502737946 >> v1:172.21.15.77:6789/0 conn(0x7f8da00d4a10 legacy=0x7f8da00b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.293+0000 7f8da6224640 1 -- 172.21.15.77:0/2502737946 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da00b5780 con 0x7f8da015e4c0 2024-04-25T01:23:36.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.293+0000 7f8da5222640 1 -- 172.21.15.77:0/2502737946 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 99386296 0 0) 0x7f8da00d1e90 con 0x7f8da00d4a10 2024-04-25T01:23:36.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.293+0000 7f8da6a25640 1 --2- 172.21.15.77:0/2502737946 >> v2:172.21.15.121:3300/0 conn(0x7f8da00d2d40 0x7f8da00cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:36.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da6224640 1 --2- 172.21.15.77:0/2502737946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8da015e4c0 0x7f8da00d2800 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8da008d2e0 tx=0x7f8d9002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8416816e32bf06d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:36.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da5222640 1 -- 172.21.15.77:0/2502737946 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d90037070 con 0x7f8da015e4c0 2024-04-25T01:23:36.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da5222640 1 -- 172.21.15.77:0/2502737946 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d90032440 con 0x7f8da015e4c0 2024-04-25T01:23:36.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da5222640 1 -- 172.21.15.77:0/2502737946 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d90032810 con 0x7f8da015e4c0 2024-04-25T01:23:36.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da8488640 1 -- 172.21.15.77:0/2502737946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8da015e4c0 msgr2=0x7f8da00d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da8488640 1 --2- 172.21.15.77:0/2502737946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8da015e4c0 0x7f8da00d2800 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8da008d2e0 tx=0x7f8d9002a1e0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da8488640 1 -- 172.21.15.77:0/2502737946 shutdown_connections 2024-04-25T01:23:36.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da8488640 1 --2- 172.21.15.77:0/2502737946 >> v2:172.21.15.121:3300/0 conn(0x7f8da00d2d40 0x7f8da00cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da8488640 1 --2- 172.21.15.77:0/2502737946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8da015e4c0 0x7f8da00d2800 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da8488640 1 -- 172.21.15.77:0/2502737946 >> 172.21.15.77:0/2502737946 conn(0x7f8da00d3ef0 msgr2=0x7f8da00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:36.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da8488640 1 -- 172.21.15.77:0/2502737946 shutdown_connections 2024-04-25T01:23:36.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.294+0000 7f8da8488640 1 -- 172.21.15.77:0/2502737946 wait complete. 2024-04-25T01:23:36.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.296+0000 7f8da8488640 1 Processor -- start 2024-04-25T01:23:36.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.296+0000 7f8da8488640 1 -- start start 2024-04-25T01:23:36.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da8488640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da00d2d40 0x7f8da00c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da8488640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8da00d4a10 0x7f8da0159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da8488640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da015e4c0 0x7f8da00b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da8488640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8da015ea20 con 0x7f8da00d4a10 2024-04-25T01:23:36.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da8488640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8da015e8a0 con 0x7f8da015e4c0 2024-04-25T01:23:36.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da8488640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8da00b45f0 con 0x7f8da00d2d40 2024-04-25T01:23:36.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da5a23640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8da00d4a10 0x7f8da0159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da5a23640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8da00d4a10 0x7f8da0159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42436/0 (socket says 172.21.15.77:42436) 2024-04-25T01:23:36.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da5a23640 1 -- 172.21.15.77:0/3700083924 learned_addr learned my addr 172.21.15.77:0/3700083924 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:36.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da6224640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da00d2d40 0x7f8da00c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da6a25640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da015e4c0 0x7f8da00b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da5a23640 1 -- 172.21.15.77:0/3700083924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da00d2d40 msgr2=0x7f8da00c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da5a23640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da00d2d40 0x7f8da00c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da5a23640 1 -- 172.21.15.77:0/3700083924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da015e4c0 msgr2=0x7f8da00b88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da5a23640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da015e4c0 0x7f8da00b88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da5a23640 1 -- 172.21.15.77:0/3700083924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da00b99b0 con 0x7f8da00d4a10 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da6224640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da00d2d40 0x7f8da00c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da6a25640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da015e4c0 0x7f8da00b88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da5a23640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8da00d4a10 0x7f8da0159d70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00c4e0 tx=0x7f8d9c00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8d96ffd640 1 -- 172.21.15.77:0/3700083924 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c018070 con 0x7f8da00d4a10 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8d96ffd640 1 -- 172.21.15.77:0/3700083924 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d9c007500 con 0x7f8da00d4a10 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.297+0000 7f8da8488640 1 -- 172.21.15.77:0/3700083924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da00b9110 con 0x7f8da00d4a10 2024-04-25T01:23:36.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.298+0000 7f8d96ffd640 1 -- 172.21.15.77:0/3700083924 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c004a20 con 0x7f8da00d4a10 2024-04-25T01:23:36.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.298+0000 7f8da8488640 1 -- 172.21.15.77:0/3700083924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da00b96d0 con 0x7f8da00d4a10 2024-04-25T01:23:36.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.298+0000 7f8da8488640 1 -- 172.21.15.77:0/3700083924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da0068fb0 con 0x7f8da00d4a10 2024-04-25T01:23:36.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.299+0000 7f8d96ffd640 1 -- 172.21.15.77:0/3700083924 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8d9c01b070 con 0x7f8da00d4a10 2024-04-25T01:23:36.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.299+0000 7f8d96ffd640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d8406be60 0x7f8d8406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.300+0000 7f8d96ffd640 1 -- 172.21.15.77:0/3700083924 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f8d9c08a260 con 0x7f8da00d4a10 2024-04-25T01:23:36.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.300+0000 7f8da6224640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d8406be60 0x7f8d8406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.300+0000 7f8da6224640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d8406be60 0x7f8d8406e320 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8d90034000 tx=0x7f8d9002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:36.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.304+0000 7f8d96ffd640 1 -- 172.21.15.77:0/3700083924 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8d9c058bd0 con 0x7f8da00d4a10 2024-04-25T01:23:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.472+0000 7f8da8488640 1 -- 172.21.15.77:0/3700083924 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8da006d760 con 0x7f8d8406be60 2024-04-25T01:23:36.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.477+0000 7f8d96ffd640 1 -- 172.21.15.77:0/3700083924 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251957 (secure 0 0 0) 0x7f8da006d760 con 0x7f8d8406be60 2024-04-25T01:23:36.461 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:23:36.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.480+0000 7f8da8488640 1 -- 172.21.15.77:0/3700083924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d8406be60 msgr2=0x7f8d8406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.480+0000 7f8da8488640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d8406be60 0x7f8d8406e320 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8d90034000 tx=0x7f8d9002e000 comp rx=0 tx=0).stop 2024-04-25T01:23:36.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.480+0000 7f8da8488640 1 -- 172.21.15.77:0/3700083924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8da00d4a10 msgr2=0x7f8da0159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.480+0000 7f8da8488640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8da00d4a10 0x7f8da0159d70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00c4e0 tx=0x7f8d9c00c9a0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.480+0000 7f8da8488640 1 -- 172.21.15.77:0/3700083924 shutdown_connections 2024-04-25T01:23:36.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.480+0000 7f8da8488640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f8d8406be60 0x7f8d8406e320 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.480+0000 7f8da8488640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8da00d4a10 0x7f8da0159d70 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.480+0000 7f8da8488640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da015e4c0 0x7f8da00b88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.480+0000 7f8da8488640 1 --2- 172.21.15.77:0/3700083924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da00d2d40 0x7f8da00c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.480+0000 7f8da8488640 1 -- 172.21.15.77:0/3700083924 >> 172.21.15.77:0/3700083924 conn(0x7f8da00d3ef0 msgr2=0x7f8da00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:36.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.481+0000 7f8da8488640 1 -- 172.21.15.77:0/3700083924 shutdown_connections 2024-04-25T01:23:36.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.481+0000 7f8da8488640 1 -- 172.21.15.77:0/3700083924 wait complete. 2024-04-25T01:23:36.472 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-04-25T01:23:35.177517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222172,"kb_used_data":5960,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752548,"statfs":{"total":773094113280,"available":772866609152,"internally_reserved":0,"allocated":6103040,"data_stored":3348948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.353712"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493248+0000","last_change":"2024-04-25T01:23:28.866357+0000","last_active":"2024-04-25T01:23:30.493248+0000","last_peered":"2024-04-25T01:23:30.493248+0000","last_clean":"2024-04-25T01:23:30.493248+0000","last_became_active":"2024-04-25T01:23:28.866046+0000","last_became_peered":"2024-04-25T01:23:28.866046+0000","last_unstale":"2024-04-25T01:23:30.493248+0000","last_undegraded":"2024-04-25T01:23:30.493248+0000","last_fullsized":"2024-04-25T01:23:30.493248+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:07.237504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493128+0000","last_change":"2024-04-25T01:23:28.863459+0000","last_active":"2024-04-25T01:23:30.493128+0000","last_peered":"2024-04-25T01:23:30.493128+0000","last_clean":"2024-04-25T01:23:30.493128+0000","last_became_active":"2024-04-25T01:23:28.863196+0000","last_became_peered":"2024-04-25T01:23:28.863196+0000","last_unstale":"2024-04-25T01:23:30.493128+0000","last_undegraded":"2024-04-25T01:23:30.493128+0000","last_fullsized":"2024-04-25T01:23:30.493128+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:58.267996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493777+0000","last_change":"2024-04-25T01:23:28.876527+0000","last_active":"2024-04-25T01:23:30.493777+0000","last_peered":"2024-04-25T01:23:30.493777+0000","last_clean":"2024-04-25T01:23:30.493777+0000","last_became_active":"2024-04-25T01:23:28.876023+0000","last_became_peered":"2024-04-25T01:23:28.876023+0000","last_unstale":"2024-04-25T01:23:30.493777+0000","last_undegraded":"2024-04-25T01:23:30.493777+0000","last_fullsized":"2024-04-25T01:23:30.493777+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:54:54.671057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493578+0000","last_change":"2024-04-25T01:23:28.869963+0000","last_active":"2024-04-25T01:23:30.493578+0000","last_peered":"2024-04-25T01:23:30.493578+0000","last_clean":"2024-04-25T01:23:30.493578+0000","last_became_active":"2024-04-25T01:23:28.869414+0000","last_became_peered":"2024-04-25T01:23:28.869414+0000","last_unstale":"2024-04-25T01:23:30.493578+0000","last_undegraded":"2024-04-25T01:23:30.493578+0000","last_fullsized":"2024-04-25T01:23:30.493578+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:04.478184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.860691+0000","last_change":"2024-04-25T01:23:28.859470+0000","last_active":"2024-04-25T01:23:29.860691+0000","last_peered":"2024-04-25T01:23:29.860691+0000","last_clean":"2024-04-25T01:23:29.860691+0000","last_became_active":"2024-04-25T01:23:28.859202+0000","last_became_peered":"2024-04-25T01:23:28.859202+0000","last_unstale":"2024-04-25T01:23:29.860691+0000","last_undegraded":"2024-04-25T01:23:29.860691+0000","last_fullsized":"2024-04-25T01:23:29.860691+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:00.152615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493226+0000","last_change":"2024-04-25T01:23:28.863503+0000","last_active":"2024-04-25T01:23:30.493226+0000","last_peered":"2024-04-25T01:23:30.493226+0000","last_clean":"2024-04-25T01:23:30.493226+0000","last_became_active":"2024-04-25T01:23:28.863253+0000","last_became_peered":"2024-04-25T01:23:28.863253+0000","last_unstale":"2024-04-25T01:23:30.493226+0000","last_undegraded":"2024-04-25T01:23:30.493226+0000","last_fullsized":"2024-04-25T01:23:30.493226+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:10.112945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492514+0000","last_change":"2024-04-25T01:23:28.875874+0000","last_active":"2024-04-25T01:23:30.492514+0000","last_peered":"2024-04-25T01:23:30.492514+0000","last_clean":"2024-04-25T01:23:30.492514+0000","last_became_active":"2024-04-25T01:23:28.875601+0000","last_became_peered":"2024-04-25T01:23:28.875601+0000","last_unstale":"2024-04-25T01:23:30.492514+0000","last_undegraded":"2024-04-25T01:23:30.492514+0000","last_fullsized":"2024-04-25T01:23:30.492514+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:16:49.826753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492667+0000","last_change":"2024-04-25T01:23:28.880101+0000","last_active":"2024-04-25T01:23:30.492667+0000","last_peered":"2024-04-25T01:23:30.492667+0000","last_clean":"2024-04-25T01:23:30.492667+0000","last_became_active":"2024-04-25T01:23:28.879839+0000","last_became_peered":"2024-04-25T01:23:28.879839+0000","last_unstale":"2024-04-25T01:23:30.492667+0000","last_undegraded":"2024-04-25T01:23:30.492667+0000","last_fullsized":"2024-04-25T01:23:30.492667+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:06:54.629272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493148+0000","last_change":"2024-04-25T01:23:28.879545+0000","last_active":"2024-04-25T01:23:30.493148+0000","last_peered":"2024-04-25T01:23:30.493148+0000","last_clean":"2024-04-25T01:23:30.493148+0000","last_became_active":"2024-04-25T01:23:28.879276+0000","last_became_peered":"2024-04-25T01:23:28.879276+0000","last_unstale":"2024-04-25T01:23:30.493148+0000","last_undegraded":"2024-04-25T01:23:30.493148+0000","last_fullsized":"2024-04-25T01:23:30.493148+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:54.456717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492873+0000","last_change":"2024-04-25T01:23:28.869034+0000","last_active":"2024-04-25T01:23:30.492873+0000","last_peered":"2024-04-25T01:23:30.492873+0000","last_clean":"2024-04-25T01:23:30.492873+0000","last_became_active":"2024-04-25T01:23:28.868798+0000","last_became_peered":"2024-04-25T01:23:28.868798+0000","last_unstale":"2024-04-25T01:23:30.492873+0000","last_undegraded":"2024-04-25T01:23:30.492873+0000","last_fullsized":"2024-04-25T01:23:30.492873+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:45.721795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.859714+0000","last_change":"2024-04-25T01:23:28.878126+0000","last_active":"2024-04-25T01:23:29.859714+0000","last_peered":"2024-04-25T01:23:29.859714+0000","last_clean":"2024-04-25T01:23:29.859714+0000","last_became_active":"2024-04-25T01:23:28.877769+0000","last_became_peered":"2024-04-25T01:23:28.877769+0000","last_unstale":"2024-04-25T01:23:29.859714+0000","last_undegraded":"2024-04-25T01:23:29.859714+0000","last_fullsized":"2024-04-25T01:23:29.859714+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:27.875540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493211+0000","last_change":"2024-04-25T01:23:28.863279+0000","last_active":"2024-04-25T01:23:30.493211+0000","last_peered":"2024-04-25T01:23:30.493211+0000","last_clean":"2024-04-25T01:23:30.493211+0000","last_became_active":"2024-04-25T01:23:28.863019+0000","last_became_peered":"2024-04-25T01:23:28.863019+0000","last_unstale":"2024-04-25T01:23:30.493211+0000","last_undegraded":"2024-04-25T01:23:30.493211+0000","last_fullsized":"2024-04-25T01:23:30.493211+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:22.411974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493070+0000","last_change":"2024-04-25T01:23:28.869143+0000","last_active":"2024-04-25T01:23:30.493070+0000","last_peered":"2024-04-25T01:23:30.493070+0000","last_clean":"2024-04-25T01:23:30.493070+0000","last_became_active":"2024-04-25T01:23:28.868906+0000","last_became_peered":"2024-04-25T01:23:28.868906+0000","last_unstale":"2024-04-25T01:23:30.493070+0000","last_undegraded":"2024-04-25T01:23:30.493070+0000","last_fullsized":"2024-04-25T01:23:30.493070+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:00:12.737723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492799+0000","last_change":"2024-04-25T01:23:28.881418+0000","last_active":"2024-04-25T01:23:30.492799+0000","last_peered":"2024-04-25T01:23:30.492799+0000","last_clean":"2024-04-25T01:23:30.492799+0000","last_became_active":"2024-04-25T01:23:28.881110+0000","last_became_peered":"2024-04-25T01:23:28.881110+0000","last_unstale":"2024-04-25T01:23:30.492799+0000","last_undegraded":"2024-04-25T01:23:30.492799+0000","last_fullsized":"2024-04-25T01:23:30.492799+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:13:11.368941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492831+0000","last_change":"2024-04-25T01:23:28.866948+0000","last_active":"2024-04-25T01:23:30.492831+0000","last_peered":"2024-04-25T01:23:30.492831+0000","last_clean":"2024-04-25T01:23:30.492831+0000","last_became_active":"2024-04-25T01:23:28.866687+0000","last_became_peered":"2024-04-25T01:23:28.866687+0000","last_unstale":"2024-04-25T01:23:30.492831+0000","last_undegraded":"2024-04-25T01:23:30.492831+0000","last_fullsized":"2024-04-25T01:23:30.492831+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:45.023561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492620+0000","last_change":"2024-04-25T01:23:28.866953+0000","last_active":"2024-04-25T01:23:30.492620+0000","last_peered":"2024-04-25T01:23:30.492620+0000","last_clean":"2024-04-25T01:23:30.492620+0000","last_became_active":"2024-04-25T01:23:28.866693+0000","last_became_peered":"2024-04-25T01:23:28.866693+0000","last_unstale":"2024-04-25T01:23:30.492620+0000","last_undegraded":"2024-04-25T01:23:30.492620+0000","last_fullsized":"2024-04-25T01:23:30.492620+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:04.439451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493111+0000","last_change":"2024-04-25T01:23:28.880212+0000","last_active":"2024-04-25T01:23:30.493111+0000","last_peered":"2024-04-25T01:23:30.493111+0000","last_clean":"2024-04-25T01:23:30.493111+0000","last_became_active":"2024-04-25T01:23:28.877955+0000","last_became_peered":"2024-04-25T01:23:28.877955+0000","last_unstale":"2024-04-25T01:23:30.493111+0000","last_undegraded":"2024-04-25T01:23:30.493111+0000","last_fullsized":"2024-04-25T01:23:30.493111+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:33:46.400136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493327+0000","last_change":"2024-04-25T01:23:28.864229+0000","last_active":"2024-04-25T01:23:30.493327+0000","last_peered":"2024-04-25T01:23:30.493327+0000","last_clean":"2024-04-25T01:23:30.493327+0000","last_became_active":"2024-04-25T01:23:28.863950+0000","last_became_peered":"2024-04-25T01:23:28.863950+0000","last_unstale":"2024-04-25T01:23:30.493327+0000","last_undegraded":"2024-04-25T01:23:30.493327+0000","last_fullsized":"2024-04-25T01:23:30.493327+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:24.366065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493862+0000","last_change":"2024-04-25T01:23:28.876300+0000","last_active":"2024-04-25T01:23:30.493862+0000","last_peered":"2024-04-25T01:23:30.493862+0000","last_clean":"2024-04-25T01:23:30.493862+0000","last_became_active":"2024-04-25T01:23:28.875651+0000","last_became_peered":"2024-04-25T01:23:28.875651+0000","last_unstale":"2024-04-25T01:23:30.493862+0000","last_undegraded":"2024-04-25T01:23:30.493862+0000","last_fullsized":"2024-04-25T01:23:30.493862+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:07:59.316699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493590+0000","last_change":"2024-04-25T01:23:28.864606+0000","last_active":"2024-04-25T01:23:30.493590+0000","last_peered":"2024-04-25T01:23:30.493590+0000","last_clean":"2024-04-25T01:23:30.493590+0000","last_became_active":"2024-04-25T01:23:28.864321+0000","last_became_peered":"2024-04-25T01:23:28.864321+0000","last_unstale":"2024-04-25T01:23:30.493590+0000","last_undegraded":"2024-04-25T01:23:30.493590+0000","last_fullsized":"2024-04-25T01:23:30.493590+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:45.266509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493151+0000","last_change":"2024-04-25T01:23:28.865430+0000","last_active":"2024-04-25T01:23:30.493151+0000","last_peered":"2024-04-25T01:23:30.493151+0000","last_clean":"2024-04-25T01:23:30.493151+0000","last_became_active":"2024-04-25T01:23:28.865137+0000","last_became_peered":"2024-04-25T01:23:28.865137+0000","last_unstale":"2024-04-25T01:23:30.493151+0000","last_undegraded":"2024-04-25T01:23:30.493151+0000","last_fullsized":"2024-04-25T01:23:30.493151+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:39.213333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492620+0000","last_change":"2024-04-25T01:23:28.878671+0000","last_active":"2024-04-25T01:23:30.492620+0000","last_peered":"2024-04-25T01:23:30.492620+0000","last_clean":"2024-04-25T01:23:30.492620+0000","last_became_active":"2024-04-25T01:23:28.878408+0000","last_became_peered":"2024-04-25T01:23:28.878408+0000","last_unstale":"2024-04-25T01:23:30.492620+0000","last_undegraded":"2024-04-25T01:23:30.492620+0000","last_fullsized":"2024-04-25T01:23:30.492620+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:43.885262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.859428+0000","last_change":"2024-04-25T01:23:28.877502+0000","last_active":"2024-04-25T01:23:29.859428+0000","last_peered":"2024-04-25T01:23:29.859428+0000","last_clean":"2024-04-25T01:23:29.859428+0000","last_became_active":"2024-04-25T01:23:28.876640+0000","last_became_peered":"2024-04-25T01:23:28.876640+0000","last_unstale":"2024-04-25T01:23:29.859428+0000","last_undegraded":"2024-04-25T01:23:29.859428+0000","last_fullsized":"2024-04-25T01:23:29.859428+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:22.318728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492295+0000","last_change":"2024-04-25T01:23:28.880758+0000","last_active":"2024-04-25T01:23:30.492295+0000","last_peered":"2024-04-25T01:23:30.492295+0000","last_clean":"2024-04-25T01:23:30.492295+0000","last_became_active":"2024-04-25T01:23:28.880490+0000","last_became_peered":"2024-04-25T01:23:28.880490+0000","last_unstale":"2024-04-25T01:23:30.492295+0000","last_undegraded":"2024-04-25T01:23:30.492295+0000","last_fullsized":"2024-04-25T01:23:30.492295+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:58.267996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493474+0000","last_change":"2024-04-25T01:23:28.880356+0000","last_active":"2024-04-25T01:23:30.493474+0000","last_peered":"2024-04-25T01:23:30.493474+0000","last_clean":"2024-04-25T01:23:30.493474+0000","last_became_active":"2024-04-25T01:23:28.876486+0000","last_became_peered":"2024-04-25T01:23:28.876486+0000","last_unstale":"2024-04-25T01:23:30.493474+0000","last_undegraded":"2024-04-25T01:23:30.493474+0000","last_fullsized":"2024-04-25T01:23:30.493474+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:43.885262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492503+0000","last_change":"2024-04-25T01:23:28.870427+0000","last_active":"2024-04-25T01:23:30.492503+0000","last_peered":"2024-04-25T01:23:30.492503+0000","last_clean":"2024-04-25T01:23:30.492503+0000","last_became_active":"2024-04-25T01:23:28.869946+0000","last_became_peered":"2024-04-25T01:23:28.869946+0000","last_unstale":"2024-04-25T01:23:30.492503+0000","last_undegraded":"2024-04-25T01:23:30.492503+0000","last_fullsized":"2024-04-25T01:23:30.492503+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:46.560872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493687+0000","last_change":"2024-04-25T01:23:28.879648+0000","last_active":"2024-04-25T01:23:30.493687+0000","last_peered":"2024-04-25T01:23:30.493687+0000","last_clean":"2024-04-25T01:23:30.493687+0000","last_became_active":"2024-04-25T01:23:28.879381+0000","last_became_peered":"2024-04-25T01:23:28.879381+0000","last_unstale":"2024-04-25T01:23:30.493687+0000","last_undegraded":"2024-04-25T01:23:30.493687+0000","last_fullsized":"2024-04-25T01:23:30.493687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:37:45.139173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493458+0000","last_change":"2024-04-25T01:23:28.866748+0000","last_active":"2024-04-25T01:23:30.493458+0000","last_peered":"2024-04-25T01:23:30.493458+0000","last_clean":"2024-04-25T01:23:30.493458+0000","last_became_active":"2024-04-25T01:23:28.866468+0000","last_became_peered":"2024-04-25T01:23:28.866468+0000","last_unstale":"2024-04-25T01:23:30.493458+0000","last_undegraded":"2024-04-25T01:23:30.493458+0000","last_fullsized":"2024-04-25T01:23:30.493458+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:29.496738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493401+0000","last_change":"2024-04-25T01:23:28.881731+0000","last_active":"2024-04-25T01:23:30.493401+0000","last_peered":"2024-04-25T01:23:30.493401+0000","last_clean":"2024-04-25T01:23:30.493401+0000","last_became_active":"2024-04-25T01:23:28.880574+0000","last_became_peered":"2024-04-25T01:23:28.880574+0000","last_unstale":"2024-04-25T01:23:30.493401+0000","last_undegraded":"2024-04-25T01:23:30.493401+0000","last_fullsized":"2024-04-25T01:23:30.493401+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:46.560872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492958+0000","last_change":"2024-04-25T01:23:28.864974+0000","last_active":"2024-04-25T01:23:30.492958+0000","last_peered":"2024-04-25T01:23:30.492958+0000","last_clean":"2024-04-25T01:23:30.492958+0000","last_became_active":"2024-04-25T01:23:28.863861+0000","last_became_peered":"2024-04-25T01:23:28.863861+0000","last_unstale":"2024-04-25T01:23:30.492958+0000","last_undegraded":"2024-04-25T01:23:30.492958+0000","last_fullsized":"2024-04-25T01:23:30.492958+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:21.843743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492573+0000","last_change":"2024-04-25T01:23:28.872091+0000","last_active":"2024-04-25T01:23:30.492573+0000","last_peered":"2024-04-25T01:23:30.492573+0000","last_clean":"2024-04-25T01:23:30.492573+0000","last_became_active":"2024-04-25T01:23:28.871816+0000","last_became_peered":"2024-04-25T01:23:28.871816+0000","last_unstale":"2024-04-25T01:23:30.492573+0000","last_undegraded":"2024-04-25T01:23:30.492573+0000","last_fullsized":"2024-04-25T01:23:30.492573+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:24.366065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.861317+0000","last_change":"2024-04-25T01:23:28.857230+0000","last_active":"2024-04-25T01:23:29.861317+0000","last_peered":"2024-04-25T01:23:29.861317+0000","last_clean":"2024-04-25T01:23:29.861317+0000","last_became_active":"2024-04-25T01:23:28.856831+0000","last_became_peered":"2024-04-25T01:23:28.856831+0000","last_unstale":"2024-04-25T01:23:29.861317+0000","last_undegraded":"2024-04-25T01:23:29.861317+0000","last_fullsized":"2024-04-25T01:23:29.861317+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:21.843743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492612+0000","last_change":"2024-04-25T01:23:28.861966+0000","last_active":"2024-04-25T01:23:30.492612+0000","last_peered":"2024-04-25T01:23:30.492612+0000","last_clean":"2024-04-25T01:23:30.492612+0000","last_became_active":"2024-04-25T01:23:28.861668+0000","last_became_peered":"2024-04-25T01:23:28.861668+0000","last_unstale":"2024-04-25T01:23:30.492612+0000","last_undegraded":"2024-04-25T01:23:30.492612+0000","last_fullsized":"2024-04-25T01:23:30.492612+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:37:45.139173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":18,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493341+0000","last_change":"2024-04-25T01:23:27.840369+0000","last_active":"2024-04-25T01:23:30.493341+0000","last_peered":"2024-04-25T01:23:30.493341+0000","last_clean":"2024-04-25T01:23:30.493341+0000","last_became_active":"2024-04-25T01:23:27.839987+0000","last_became_peered":"2024-04-25T01:23:27.839987+0000","last_unstale":"2024-04-25T01:23:30.493341+0000","last_undegraded":"2024-04-25T01:23:30.493341+0000","last_fullsized":"2024-04-25T01:23:30.493341+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:40.061850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.860468+0000","last_change":"2024-04-25T01:23:28.863487+0000","last_active":"2024-04-25T01:23:29.860468+0000","last_peered":"2024-04-25T01:23:29.860468+0000","last_clean":"2024-04-25T01:23:29.860468+0000","last_became_active":"2024-04-25T01:23:28.862841+0000","last_became_peered":"2024-04-25T01:23:28.862841+0000","last_unstale":"2024-04-25T01:23:29.860468+0000","last_undegraded":"2024-04-25T01:23:29.860468+0000","last_fullsized":"2024-04-25T01:23:29.860468+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:45.023561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492957+0000","last_change":"2024-04-25T01:23:27.838394+0000","last_active":"2024-04-25T01:23:30.492957+0000","last_peered":"2024-04-25T01:23:30.492957+0000","last_clean":"2024-04-25T01:23:30.492957+0000","last_became_active":"2024-04-25T01:23:27.838114+0000","last_became_peered":"2024-04-25T01:23:27.838114+0000","last_unstale":"2024-04-25T01:23:30.492957+0000","last_undegraded":"2024-04-25T01:23:30.492957+0000","last_fullsized":"2024-04-25T01:23:30.492957+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:56.394674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493333+0000","last_change":"2024-04-25T01:23:28.864771+0000","last_active":"2024-04-25T01:23:30.493333+0000","last_peered":"2024-04-25T01:23:30.493333+0000","last_clean":"2024-04-25T01:23:30.493333+0000","last_became_active":"2024-04-25T01:23:28.864332+0000","last_became_peered":"2024-04-25T01:23:28.864332+0000","last_unstale":"2024-04-25T01:23:30.493333+0000","last_undegraded":"2024-04-25T01:23:30.493333+0000","last_fullsized":"2024-04-25T01:23:30.493333+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:14.293718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493719+0000","last_change":"2024-04-25T01:23:27.839860+0000","last_active":"2024-04-25T01:23:30.493719+0000","last_peered":"2024-04-25T01:23:30.493719+0000","last_clean":"2024-04-25T01:23:30.493719+0000","last_became_active":"2024-04-25T01:23:27.839589+0000","last_became_peered":"2024-04-25T01:23:27.839589+0000","last_unstale":"2024-04-25T01:23:30.493719+0000","last_undegraded":"2024-04-25T01:23:30.493719+0000","last_fullsized":"2024-04-25T01:23:30.493719+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:08.739387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492866+0000","last_change":"2024-04-25T01:23:28.870964+0000","last_active":"2024-04-25T01:23:30.492866+0000","last_peered":"2024-04-25T01:23:30.492866+0000","last_clean":"2024-04-25T01:23:30.492866+0000","last_became_active":"2024-04-25T01:23:28.868769+0000","last_became_peered":"2024-04-25T01:23:28.868769+0000","last_unstale":"2024-04-25T01:23:30.492866+0000","last_undegraded":"2024-04-25T01:23:30.492866+0000","last_fullsized":"2024-04-25T01:23:30.492866+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:37:37.009175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.861781+0000","last_change":"2024-04-25T01:23:27.842142+0000","last_active":"2024-04-25T01:23:29.861781+0000","last_peered":"2024-04-25T01:23:29.861781+0000","last_clean":"2024-04-25T01:23:29.861781+0000","last_became_active":"2024-04-25T01:23:27.841740+0000","last_became_peered":"2024-04-25T01:23:27.841740+0000","last_unstale":"2024-04-25T01:23:29.861781+0000","last_undegraded":"2024-04-25T01:23:29.861781+0000","last_fullsized":"2024-04-25T01:23:29.861781+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:44.264978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.860615+0000","last_change":"2024-04-25T01:23:28.872238+0000","last_active":"2024-04-25T01:23:29.860615+0000","last_peered":"2024-04-25T01:23:29.860615+0000","last_clean":"2024-04-25T01:23:29.860615+0000","last_became_active":"2024-04-25T01:23:28.871901+0000","last_became_peered":"2024-04-25T01:23:28.871901+0000","last_unstale":"2024-04-25T01:23:29.860615+0000","last_undegraded":"2024-04-25T01:23:29.860615+0000","last_fullsized":"2024-04-25T01:23:29.860615+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:27.979663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493485+0000","last_change":"2024-04-25T01:23:27.831214+0000","last_active":"2024-04-25T01:23:30.493485+0000","last_peered":"2024-04-25T01:23:30.493485+0000","last_clean":"2024-04-25T01:23:30.493485+0000","last_became_active":"2024-04-25T01:23:27.830883+0000","last_became_peered":"2024-04-25T01:23:27.830883+0000","last_unstale":"2024-04-25T01:23:30.493485+0000","last_undegraded":"2024-04-25T01:23:30.493485+0000","last_fullsized":"2024-04-25T01:23:30.493485+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:59.151083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493074+0000","last_change":"2024-04-25T01:23:28.864866+0000","last_active":"2024-04-25T01:23:30.493074+0000","last_peered":"2024-04-25T01:23:30.493074+0000","last_clean":"2024-04-25T01:23:30.493074+0000","last_became_active":"2024-04-25T01:23:28.864372+0000","last_became_peered":"2024-04-25T01:23:28.864372+0000","last_unstale":"2024-04-25T01:23:30.493074+0000","last_undegraded":"2024-04-25T01:23:30.493074+0000","last_fullsized":"2024-04-25T01:23:30.493074+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:59:22.604489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.860395+0000","last_change":"2024-04-25T01:23:27.839506+0000","last_active":"2024-04-25T01:23:29.860395+0000","last_peered":"2024-04-25T01:23:29.860395+0000","last_clean":"2024-04-25T01:23:29.860395+0000","last_became_active":"2024-04-25T01:23:27.838600+0000","last_became_peered":"2024-04-25T01:23:27.838600+0000","last_unstale":"2024-04-25T01:23:29.860395+0000","last_undegraded":"2024-04-25T01:23:29.860395+0000","last_fullsized":"2024-04-25T01:23:29.860395+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:26:54.800425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492844+0000","last_change":"2024-04-25T01:23:28.870219+0000","last_active":"2024-04-25T01:23:30.492844+0000","last_peered":"2024-04-25T01:23:30.492844+0000","last_clean":"2024-04-25T01:23:30.492844+0000","last_became_active":"2024-04-25T01:23:28.869622+0000","last_became_peered":"2024-04-25T01:23:28.869622+0000","last_unstale":"2024-04-25T01:23:30.492844+0000","last_undegraded":"2024-04-25T01:23:30.492844+0000","last_fullsized":"2024-04-25T01:23:30.492844+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:06.023658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492919+0000","last_change":"2024-04-25T01:23:27.843276+0000","last_active":"2024-04-25T01:23:30.492919+0000","last_peered":"2024-04-25T01:23:30.492919+0000","last_clean":"2024-04-25T01:23:30.492919+0000","last_became_active":"2024-04-25T01:23:27.842824+0000","last_became_peered":"2024-04-25T01:23:27.842824+0000","last_unstale":"2024-04-25T01:23:30.492919+0000","last_undegraded":"2024-04-25T01:23:30.492919+0000","last_fullsized":"2024-04-25T01:23:30.492919+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:52.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493396+0000","last_change":"2024-04-25T01:23:28.876301+0000","last_active":"2024-04-25T01:23:30.493396+0000","last_peered":"2024-04-25T01:23:30.493396+0000","last_clean":"2024-04-25T01:23:30.493396+0000","last_became_active":"2024-04-25T01:23:28.876006+0000","last_became_peered":"2024-04-25T01:23:28.876006+0000","last_unstale":"2024-04-25T01:23:30.493396+0000","last_undegraded":"2024-04-25T01:23:30.493396+0000","last_fullsized":"2024-04-25T01:23:30.493396+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:16:49.826753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492317+0000","last_change":"2024-04-25T01:23:27.842530+0000","last_active":"2024-04-25T01:23:30.492317+0000","last_peered":"2024-04-25T01:23:30.492317+0000","last_clean":"2024-04-25T01:23:30.492317+0000","last_became_active":"2024-04-25T01:23:27.842265+0000","last_became_peered":"2024-04-25T01:23:27.842265+0000","last_unstale":"2024-04-25T01:23:30.492317+0000","last_undegraded":"2024-04-25T01:23:30.492317+0000","last_fullsized":"2024-04-25T01:23:30.492317+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:01.477667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.860189+0000","last_change":"2024-04-25T01:23:28.856134+0000","last_active":"2024-04-25T01:23:29.860189+0000","last_peered":"2024-04-25T01:23:29.860189+0000","last_clean":"2024-04-25T01:23:29.860189+0000","last_became_active":"2024-04-25T01:23:28.855854+0000","last_became_peered":"2024-04-25T01:23:28.855854+0000","last_unstale":"2024-04-25T01:23:29.860189+0000","last_undegraded":"2024-04-25T01:23:29.860189+0000","last_fullsized":"2024-04-25T01:23:29.860189+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:28.001620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493495+0000","last_change":"2024-04-25T01:23:27.842714+0000","last_active":"2024-04-25T01:23:30.493495+0000","last_peered":"2024-04-25T01:23:30.493495+0000","last_clean":"2024-04-25T01:23:30.493495+0000","last_became_active":"2024-04-25T01:23:27.842268+0000","last_became_peered":"2024-04-25T01:23:27.842268+0000","last_unstale":"2024-04-25T01:23:30.493495+0000","last_undegraded":"2024-04-25T01:23:30.493495+0000","last_fullsized":"2024-04-25T01:23:30.493495+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:44.264978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.860325+0000","last_change":"2024-04-25T01:23:28.859619+0000","last_active":"2024-04-25T01:23:29.860325+0000","last_peered":"2024-04-25T01:23:29.860325+0000","last_clean":"2024-04-25T01:23:29.860325+0000","last_became_active":"2024-04-25T01:23:28.859352+0000","last_became_peered":"2024-04-25T01:23:28.859352+0000","last_unstale":"2024-04-25T01:23:29.860325+0000","last_undegraded":"2024-04-25T01:23:29.860325+0000","last_fullsized":"2024-04-25T01:23:29.860325+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:37:45.139173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.860734+0000","last_change":"2024-04-25T01:23:27.839254+0000","last_active":"2024-04-25T01:23:29.860734+0000","last_peered":"2024-04-25T01:23:29.860734+0000","last_clean":"2024-04-25T01:23:29.860734+0000","last_became_active":"2024-04-25T01:23:27.838972+0000","last_became_peered":"2024-04-25T01:23:27.838972+0000","last_unstale":"2024-04-25T01:23:29.860734+0000","last_undegraded":"2024-04-25T01:23:29.860734+0000","last_fullsized":"2024-04-25T01:23:29.860734+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:40.061850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493035+0000","last_change":"2024-04-25T01:23:28.877408+0000","last_active":"2024-04-25T01:23:30.493035+0000","last_peered":"2024-04-25T01:23:30.493035+0000","last_clean":"2024-04-25T01:23:30.493035+0000","last_became_active":"2024-04-25T01:23:28.877140+0000","last_became_peered":"2024-04-25T01:23:28.877140+0000","last_unstale":"2024-04-25T01:23:30.493035+0000","last_undegraded":"2024-04-25T01:23:30.493035+0000","last_fullsized":"2024-04-25T01:23:30.493035+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:50.391761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":17,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493297+0000","last_change":"2024-04-25T01:23:27.843358+0000","last_active":"2024-04-25T01:23:30.493297+0000","last_peered":"2024-04-25T01:23:30.493297+0000","last_clean":"2024-04-25T01:23:30.493297+0000","last_became_active":"2024-04-25T01:23:27.843098+0000","last_became_peered":"2024-04-25T01:23:27.843098+0000","last_unstale":"2024-04-25T01:23:30.493297+0000","last_undegraded":"2024-04-25T01:23:30.493297+0000","last_fullsized":"2024-04-25T01:23:30.493297+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:36:13.292186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.860812+0000","last_change":"2024-04-25T01:23:28.868985+0000","last_active":"2024-04-25T01:23:29.860812+0000","last_peered":"2024-04-25T01:23:29.860812+0000","last_clean":"2024-04-25T01:23:29.860812+0000","last_became_active":"2024-04-25T01:23:28.868761+0000","last_became_peered":"2024-04-25T01:23:28.868761+0000","last_unstale":"2024-04-25T01:23:29.860812+0000","last_undegraded":"2024-04-25T01:23:29.860812+0000","last_fullsized":"2024-04-25T01:23:29.860812+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:02.479199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":20,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493297+0000","last_change":"2024-04-25T01:23:27.841668+0000","last_active":"2024-04-25T01:23:30.493297+0000","last_peered":"2024-04-25T01:23:30.493297+0000","last_clean":"2024-04-25T01:23:30.493297+0000","last_became_active":"2024-04-25T01:23:27.841382+0000","last_became_peered":"2024-04-25T01:23:27.841382+0000","last_unstale":"2024-04-25T01:23:30.493297+0000","last_undegraded":"2024-04-25T01:23:30.493297+0000","last_fullsized":"2024-04-25T01:23:30.493297+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:37:44.137641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.861681+0000","last_change":"2024-04-25T01:23:28.862937+0000","last_active":"2024-04-25T01:23:29.861681+0000","last_peered":"2024-04-25T01:23:29.861681+0000","last_clean":"2024-04-25T01:23:29.861681+0000","last_became_active":"2024-04-25T01:23:28.862600+0000","last_became_peered":"2024-04-25T01:23:28.862600+0000","last_unstale":"2024-04-25T01:23:29.861681+0000","last_undegraded":"2024-04-25T01:23:29.861681+0000","last_fullsized":"2024-04-25T01:23:29.861681+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:16:49.826753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493574+0000","last_change":"2024-04-25T01:23:27.845237+0000","last_active":"2024-04-25T01:23:30.493574+0000","last_peered":"2024-04-25T01:23:30.493574+0000","last_clean":"2024-04-25T01:23:30.493574+0000","last_became_active":"2024-04-25T01:23:27.844336+0000","last_became_peered":"2024-04-25T01:23:27.844336+0000","last_unstale":"2024-04-25T01:23:30.493574+0000","last_undegraded":"2024-04-25T01:23:30.493574+0000","last_fullsized":"2024-04-25T01:23:30.493574+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:09.155513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493923+0000","last_change":"2024-04-25T01:23:28.878910+0000","last_active":"2024-04-25T01:23:30.493923+0000","last_peered":"2024-04-25T01:23:30.493923+0000","last_clean":"2024-04-25T01:23:30.493923+0000","last_became_active":"2024-04-25T01:23:28.878643+0000","last_became_peered":"2024-04-25T01:23:28.878643+0000","last_unstale":"2024-04-25T01:23:30.493923+0000","last_undegraded":"2024-04-25T01:23:30.493923+0000","last_fullsized":"2024-04-25T01:23:30.493923+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:06.023658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493186+0000","last_change":"2024-04-25T01:23:27.848962+0000","last_active":"2024-04-25T01:23:30.493186+0000","last_peered":"2024-04-25T01:23:30.493186+0000","last_clean":"2024-04-25T01:23:30.493186+0000","last_became_active":"2024-04-25T01:23:27.848694+0000","last_became_peered":"2024-04-25T01:23:27.848694+0000","last_unstale":"2024-04-25T01:23:30.493186+0000","last_undegraded":"2024-04-25T01:23:30.493186+0000","last_fullsized":"2024-04-25T01:23:30.493186+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:26:54.800425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492715+0000","last_change":"2024-04-25T01:23:28.862982+0000","last_active":"2024-04-25T01:23:30.492715+0000","last_peered":"2024-04-25T01:23:30.492715+0000","last_clean":"2024-04-25T01:23:30.492715+0000","last_became_active":"2024-04-25T01:23:28.862683+0000","last_became_peered":"2024-04-25T01:23:28.862683+0000","last_unstale":"2024-04-25T01:23:30.492715+0000","last_undegraded":"2024-04-25T01:23:30.492715+0000","last_fullsized":"2024-04-25T01:23:30.492715+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:07:59.316699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492789+0000","last_change":"2024-04-25T01:23:27.843254+0000","last_active":"2024-04-25T01:23:30.492789+0000","last_peered":"2024-04-25T01:23:30.492789+0000","last_clean":"2024-04-25T01:23:30.492789+0000","last_became_active":"2024-04-25T01:23:27.842506+0000","last_became_peered":"2024-04-25T01:23:27.842506+0000","last_unstale":"2024-04-25T01:23:30.492789+0000","last_undegraded":"2024-04-25T01:23:30.492789+0000","last_fullsized":"2024-04-25T01:23:30.492789+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:09.155513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493330+0000","last_change":"2024-04-25T01:23:28.879997+0000","last_active":"2024-04-25T01:23:30.493330+0000","last_peered":"2024-04-25T01:23:30.493330+0000","last_clean":"2024-04-25T01:23:30.493330+0000","last_became_active":"2024-04-25T01:23:28.877445+0000","last_became_peered":"2024-04-25T01:23:28.877445+0000","last_unstale":"2024-04-25T01:23:30.493330+0000","last_undegraded":"2024-04-25T01:23:30.493330+0000","last_fullsized":"2024-04-25T01:23:30.493330+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:42:11.551594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":22,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492753+0000","last_change":"2024-04-25T01:23:27.841058+0000","last_active":"2024-04-25T01:23:30.492753+0000","last_peered":"2024-04-25T01:23:30.492753+0000","last_clean":"2024-04-25T01:23:30.492753+0000","last_became_active":"2024-04-25T01:23:27.840808+0000","last_became_peered":"2024-04-25T01:23:27.840808+0000","last_unstale":"2024-04-25T01:23:30.492753+0000","last_undegraded":"2024-04-25T01:23:30.492753+0000","last_fullsized":"2024-04-25T01:23:30.492753+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:06.235972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492684+0000","last_change":"2024-04-25T01:23:28.855912+0000","last_active":"2024-04-25T01:23:30.492684+0000","last_peered":"2024-04-25T01:23:30.492684+0000","last_clean":"2024-04-25T01:23:30.492684+0000","last_became_active":"2024-04-25T01:23:28.855599+0000","last_became_peered":"2024-04-25T01:23:28.855599+0000","last_unstale":"2024-04-25T01:23:30.492684+0000","last_undegraded":"2024-04-25T01:23:30.492684+0000","last_fullsized":"2024-04-25T01:23:30.492684+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:54:54.671057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493554+0000","last_change":"2024-04-25T01:23:27.846019+0000","last_active":"2024-04-25T01:23:30.493554+0000","last_peered":"2024-04-25T01:23:30.493554+0000","last_clean":"2024-04-25T01:23:30.493554+0000","last_became_active":"2024-04-25T01:23:27.845056+0000","last_became_peered":"2024-04-25T01:23:27.845056+0000","last_unstale":"2024-04-25T01:23:30.493554+0000","last_undegraded":"2024-04-25T01:23:30.493554+0000","last_fullsized":"2024-04-25T01:23:30.493554+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:42.248852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493787+0000","last_change":"2024-04-25T01:23:28.873143+0000","last_active":"2024-04-25T01:23:30.493787+0000","last_peered":"2024-04-25T01:23:30.493787+0000","last_clean":"2024-04-25T01:23:30.493787+0000","last_became_active":"2024-04-25T01:23:28.872882+0000","last_became_peered":"2024-04-25T01:23:28.872882+0000","last_unstale":"2024-04-25T01:23:30.493787+0000","last_undegraded":"2024-04-25T01:23:30.493787+0000","last_fullsized":"2024-04-25T01:23:30.493787+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:45.023561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.859709+0000","last_change":"2024-04-25T01:23:27.839989+0000","last_active":"2024-04-25T01:23:29.859709+0000","last_peered":"2024-04-25T01:23:29.859709+0000","last_clean":"2024-04-25T01:23:29.859709+0000","last_became_active":"2024-04-25T01:23:27.839737+0000","last_became_peered":"2024-04-25T01:23:27.839737+0000","last_unstale":"2024-04-25T01:23:29.859709+0000","last_undegraded":"2024-04-25T01:23:29.859709+0000","last_fullsized":"2024-04-25T01:23:29.859709+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:43.042472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.860314+0000","last_change":"2024-04-25T01:23:28.872331+0000","last_active":"2024-04-25T01:23:29.860314+0000","last_peered":"2024-04-25T01:23:29.860314+0000","last_clean":"2024-04-25T01:23:29.860314+0000","last_became_active":"2024-04-25T01:23:28.871967+0000","last_became_peered":"2024-04-25T01:23:28.871967+0000","last_unstale":"2024-04-25T01:23:29.860314+0000","last_undegraded":"2024-04-25T01:23:29.860314+0000","last_fullsized":"2024-04-25T01:23:29.860314+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:57.396206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493428+0000","last_change":"2024-04-25T01:23:27.838563+0000","last_active":"2024-04-25T01:23:30.493428+0000","last_peered":"2024-04-25T01:23:30.493428+0000","last_clean":"2024-04-25T01:23:30.493428+0000","last_became_active":"2024-04-25T01:23:27.837764+0000","last_became_peered":"2024-04-25T01:23:27.837764+0000","last_unstale":"2024-04-25T01:23:30.493428+0000","last_undegraded":"2024-04-25T01:23:30.493428+0000","last_fullsized":"2024-04-25T01:23:30.493428+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:48.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493125+0000","last_change":"2024-04-25T01:23:28.872372+0000","last_active":"2024-04-25T01:23:30.493125+0000","last_peered":"2024-04-25T01:23:30.493125+0000","last_clean":"2024-04-25T01:23:30.493125+0000","last_became_active":"2024-04-25T01:23:28.871873+0000","last_became_peered":"2024-04-25T01:23:28.871873+0000","last_unstale":"2024-04-25T01:23:30.493125+0000","last_undegraded":"2024-04-25T01:23:30.493125+0000","last_fullsized":"2024-04-25T01:23:30.493125+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:23.707080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493562+0000","last_change":"2024-04-25T01:23:27.843101+0000","last_active":"2024-04-25T01:23:30.493562+0000","last_peered":"2024-04-25T01:23:30.493562+0000","last_clean":"2024-04-25T01:23:30.493562+0000","last_became_active":"2024-04-25T01:23:27.842810+0000","last_became_peered":"2024-04-25T01:23:27.842810+0000","last_unstale":"2024-04-25T01:23:30.493562+0000","last_undegraded":"2024-04-25T01:23:30.493562+0000","last_fullsized":"2024-04-25T01:23:30.493562+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:33.027603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.861791+0000","last_change":"2024-04-25T01:23:28.860352+0000","last_active":"2024-04-25T01:23:29.861791+0000","last_peered":"2024-04-25T01:23:29.861791+0000","last_clean":"2024-04-25T01:23:29.861791+0000","last_became_active":"2024-04-25T01:23:28.860103+0000","last_became_peered":"2024-04-25T01:23:28.860103+0000","last_unstale":"2024-04-25T01:23:29.861791+0000","last_undegraded":"2024-04-25T01:23:29.861791+0000","last_fullsized":"2024-04-25T01:23:29.861791+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:24.366065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493166+0000","last_change":"2024-04-25T01:23:27.834285+0000","last_active":"2024-04-25T01:23:30.493166+0000","last_peered":"2024-04-25T01:23:30.493166+0000","last_clean":"2024-04-25T01:23:30.493166+0000","last_became_active":"2024-04-25T01:23:27.834030+0000","last_became_peered":"2024-04-25T01:23:27.834030+0000","last_unstale":"2024-04-25T01:23:30.493166+0000","last_undegraded":"2024-04-25T01:23:30.493166+0000","last_fullsized":"2024-04-25T01:23:30.493166+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:27.000088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493321+0000","last_change":"2024-04-25T01:23:28.864952+0000","last_active":"2024-04-25T01:23:30.493321+0000","last_peered":"2024-04-25T01:23:30.493321+0000","last_clean":"2024-04-25T01:23:30.493321+0000","last_became_active":"2024-04-25T01:23:28.864659+0000","last_became_peered":"2024-04-25T01:23:28.864659+0000","last_unstale":"2024-04-25T01:23:30.493321+0000","last_undegraded":"2024-04-25T01:23:30.493321+0000","last_fullsized":"2024-04-25T01:23:30.493321+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:10.157045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492581+0000","last_change":"2024-04-25T01:23:27.839744+0000","last_active":"2024-04-25T01:23:30.492581+0000","last_peered":"2024-04-25T01:23:30.492581+0000","last_clean":"2024-04-25T01:23:30.492581+0000","last_became_active":"2024-04-25T01:23:27.839478+0000","last_became_peered":"2024-04-25T01:23:27.839478+0000","last_unstale":"2024-04-25T01:23:30.492581+0000","last_undegraded":"2024-04-25T01:23:30.492581+0000","last_fullsized":"2024-04-25T01:23:30.492581+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:48.513923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493542+0000","last_change":"2024-04-25T01:23:28.879949+0000","last_active":"2024-04-25T01:23:30.493542+0000","last_peered":"2024-04-25T01:23:30.493542+0000","last_clean":"2024-04-25T01:23:30.493542+0000","last_became_active":"2024-04-25T01:23:28.877487+0000","last_became_peered":"2024-04-25T01:23:28.877487+0000","last_unstale":"2024-04-25T01:23:30.493542+0000","last_undegraded":"2024-04-25T01:23:30.493542+0000","last_fullsized":"2024-04-25T01:23:30.493542+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:06:54.629272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492653+0000","last_change":"2024-04-25T01:23:27.843276+0000","last_active":"2024-04-25T01:23:30.492653+0000","last_peered":"2024-04-25T01:23:30.492653+0000","last_clean":"2024-04-25T01:23:30.492653+0000","last_became_active":"2024-04-25T01:23:27.842913+0000","last_became_peered":"2024-04-25T01:23:27.842913+0000","last_unstale":"2024-04-25T01:23:30.492653+0000","last_undegraded":"2024-04-25T01:23:30.492653+0000","last_fullsized":"2024-04-25T01:23:30.492653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:56.394674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493261+0000","last_change":"2024-04-25T01:23:28.868045+0000","last_active":"2024-04-25T01:23:30.493261+0000","last_peered":"2024-04-25T01:23:30.493261+0000","last_clean":"2024-04-25T01:23:30.493261+0000","last_became_active":"2024-04-25T01:23:28.867786+0000","last_became_peered":"2024-04-25T01:23:28.867786+0000","last_unstale":"2024-04-25T01:23:30.493261+0000","last_undegraded":"2024-04-25T01:23:30.493261+0000","last_fullsized":"2024-04-25T01:23:30.493261+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:23.763653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.859357+0000","last_change":"2024-04-25T01:23:27.843723+0000","last_active":"2024-04-25T01:23:29.859357+0000","last_peered":"2024-04-25T01:23:29.859357+0000","last_clean":"2024-04-25T01:23:29.859357+0000","last_became_active":"2024-04-25T01:23:27.843464+0000","last_became_peered":"2024-04-25T01:23:27.843464+0000","last_unstale":"2024-04-25T01:23:29.859357+0000","last_undegraded":"2024-04-25T01:23:29.859357+0000","last_fullsized":"2024-04-25T01:23:29.859357+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:54.288649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.861488+0000","last_change":"2024-04-25T01:23:28.856337+0000","last_active":"2024-04-25T01:23:29.861488+0000","last_peered":"2024-04-25T01:23:29.861488+0000","last_clean":"2024-04-25T01:23:29.861488+0000","last_became_active":"2024-04-25T01:23:28.856090+0000","last_became_peered":"2024-04-25T01:23:28.856090+0000","last_unstale":"2024-04-25T01:23:29.861488+0000","last_undegraded":"2024-04-25T01:23:29.861488+0000","last_fullsized":"2024-04-25T01:23:29.861488+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:59:22.604489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493295+0000","last_change":"2024-04-25T01:23:27.847328+0000","last_active":"2024-04-25T01:23:30.493295+0000","last_peered":"2024-04-25T01:23:30.493295+0000","last_clean":"2024-04-25T01:23:30.493295+0000","last_became_active":"2024-04-25T01:23:27.847046+0000","last_became_peered":"2024-04-25T01:23:27.847046+0000","last_unstale":"2024-04-25T01:23:30.493295+0000","last_undegraded":"2024-04-25T01:23:30.493295+0000","last_fullsized":"2024-04-25T01:23:30.493295+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:49:28.578991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493370+0000","last_change":"2024-04-25T01:23:28.866850+0000","last_active":"2024-04-25T01:23:30.493370+0000","last_peered":"2024-04-25T01:23:30.493370+0000","last_clean":"2024-04-25T01:23:30.493370+0000","last_became_active":"2024-04-25T01:23:28.866592+0000","last_became_peered":"2024-04-25T01:23:28.866592+0000","last_unstale":"2024-04-25T01:23:30.493370+0000","last_undegraded":"2024-04-25T01:23:30.493370+0000","last_fullsized":"2024-04-25T01:23:30.493370+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:49.515455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493415+0000","last_change":"2024-04-25T01:23:27.842809+0000","last_active":"2024-04-25T01:23:30.493415+0000","last_peered":"2024-04-25T01:23:30.493415+0000","last_clean":"2024-04-25T01:23:30.493415+0000","last_became_active":"2024-04-25T01:23:27.842561+0000","last_became_peered":"2024-04-25T01:23:27.842561+0000","last_unstale":"2024-04-25T01:23:30.493415+0000","last_undegraded":"2024-04-25T01:23:30.493415+0000","last_fullsized":"2024-04-25T01:23:30.493415+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:28.495206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492892+0000","last_change":"2024-04-25T01:23:28.873216+0000","last_active":"2024-04-25T01:23:30.492892+0000","last_peered":"2024-04-25T01:23:30.492892+0000","last_clean":"2024-04-25T01:23:30.492892+0000","last_became_active":"2024-04-25T01:23:28.872898+0000","last_became_peered":"2024-04-25T01:23:28.872898+0000","last_unstale":"2024-04-25T01:23:30.492892+0000","last_undegraded":"2024-04-25T01:23:30.492892+0000","last_fullsized":"2024-04-25T01:23:30.492892+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:20:45.791470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492857+0000","last_change":"2024-04-25T01:23:27.844808+0000","last_active":"2024-04-25T01:23:30.492857+0000","last_peered":"2024-04-25T01:23:30.492857+0000","last_clean":"2024-04-25T01:23:30.492857+0000","last_became_active":"2024-04-25T01:23:27.844541+0000","last_became_peered":"2024-04-25T01:23:27.844541+0000","last_unstale":"2024-04-25T01:23:30.492857+0000","last_undegraded":"2024-04-25T01:23:30.492857+0000","last_fullsized":"2024-04-25T01:23:30.492857+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:26.978132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492717+0000","last_change":"2024-04-25T01:23:28.876686+0000","last_active":"2024-04-25T01:23:30.492717+0000","last_peered":"2024-04-25T01:23:30.492717+0000","last_clean":"2024-04-25T01:23:30.492717+0000","last_became_active":"2024-04-25T01:23:28.875990+0000","last_became_peered":"2024-04-25T01:23:28.875990+0000","last_unstale":"2024-04-25T01:23:30.492717+0000","last_undegraded":"2024-04-25T01:23:30.492717+0000","last_fullsized":"2024-04-25T01:23:30.492717+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:43:04.478184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492943+0000","last_change":"2024-04-25T01:23:27.842454+0000","last_active":"2024-04-25T01:23:30.492943+0000","last_peered":"2024-04-25T01:23:30.492943+0000","last_clean":"2024-04-25T01:23:30.492943+0000","last_became_active":"2024-04-25T01:23:27.842209+0000","last_became_peered":"2024-04-25T01:23:27.842209+0000","last_unstale":"2024-04-25T01:23:30.492943+0000","last_undegraded":"2024-04-25T01:23:30.492943+0000","last_fullsized":"2024-04-25T01:23:30.492943+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:09.111414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.861506+0000","last_change":"2024-04-25T01:23:28.863257+0000","last_active":"2024-04-25T01:23:29.861506+0000","last_peered":"2024-04-25T01:23:29.861506+0000","last_clean":"2024-04-25T01:23:29.861506+0000","last_became_active":"2024-04-25T01:23:28.862945+0000","last_became_peered":"2024-04-25T01:23:28.862945+0000","last_unstale":"2024-04-25T01:23:29.861506+0000","last_undegraded":"2024-04-25T01:23:29.861506+0000","last_fullsized":"2024-04-25T01:23:29.861506+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:58.267996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492687+0000","last_change":"2024-04-25T01:23:27.841080+0000","last_active":"2024-04-25T01:23:30.492687+0000","last_peered":"2024-04-25T01:23:30.492687+0000","last_clean":"2024-04-25T01:23:30.492687+0000","last_became_active":"2024-04-25T01:23:27.840813+0000","last_became_peered":"2024-04-25T01:23:27.840813+0000","last_unstale":"2024-04-25T01:23:30.492687+0000","last_undegraded":"2024-04-25T01:23:30.492687+0000","last_fullsized":"2024-04-25T01:23:30.492687+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:22.762121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493332+0000","last_change":"2024-04-25T01:23:28.880427+0000","last_active":"2024-04-25T01:23:30.493332+0000","last_peered":"2024-04-25T01:23:30.493332+0000","last_clean":"2024-04-25T01:23:30.493332+0000","last_became_active":"2024-04-25T01:23:28.880160+0000","last_became_peered":"2024-04-25T01:23:28.880160+0000","last_unstale":"2024-04-25T01:23:30.493332+0000","last_undegraded":"2024-04-25T01:23:30.493332+0000","last_fullsized":"2024-04-25T01:23:30.493332+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:19.969020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.860218+0000","last_change":"2024-04-25T01:23:27.841167+0000","last_active":"2024-04-25T01:23:29.860218+0000","last_peered":"2024-04-25T01:23:29.860218+0000","last_clean":"2024-04-25T01:23:29.860218+0000","last_became_active":"2024-04-25T01:23:27.840932+0000","last_became_peered":"2024-04-25T01:23:27.840932+0000","last_unstale":"2024-04-25T01:23:29.860218+0000","last_undegraded":"2024-04-25T01:23:29.860218+0000","last_fullsized":"2024-04-25T01:23:29.860218+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:04.282731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492843+0000","last_change":"2024-04-25T01:23:28.869091+0000","last_active":"2024-04-25T01:23:30.492843+0000","last_peered":"2024-04-25T01:23:30.492843+0000","last_clean":"2024-04-25T01:23:30.492843+0000","last_became_active":"2024-04-25T01:23:28.868833+0000","last_became_peered":"2024-04-25T01:23:28.868833+0000","last_unstale":"2024-04-25T01:23:30.492843+0000","last_undegraded":"2024-04-25T01:23:30.492843+0000","last_fullsized":"2024-04-25T01:23:30.492843+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:56:07.701449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:29.861586+0000","last_change":"2024-04-25T01:23:27.838507+0000","last_active":"2024-04-25T01:23:29.861586+0000","last_peered":"2024-04-25T01:23:29.861586+0000","last_clean":"2024-04-25T01:23:29.861586+0000","last_became_active":"2024-04-25T01:23:27.838016+0000","last_became_peered":"2024-04-25T01:23:27.838016+0000","last_unstale":"2024-04-25T01:23:29.861586+0000","last_undegraded":"2024-04-25T01:23:29.861586+0000","last_fullsized":"2024-04-25T01:23:29.861586+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:03.437919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492882+0000","last_change":"2024-04-25T01:23:28.865893+0000","last_active":"2024-04-25T01:23:30.492882+0000","last_peered":"2024-04-25T01:23:30.492882+0000","last_clean":"2024-04-25T01:23:30.492882+0000","last_became_active":"2024-04-25T01:23:28.865612+0000","last_became_peered":"2024-04-25T01:23:28.865612+0000","last_unstale":"2024-04-25T01:23:30.492882+0000","last_undegraded":"2024-04-25T01:23:30.492882+0000","last_fullsized":"2024-04-25T01:23:30.492882+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:27.818947+0000","last_clean_scrub_stamp":"2024-04-25T01:23:27.818947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:43.250384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.493683+0000","last_change":"2024-04-25T01:23:27.841576+0000","last_active":"2024-04-25T01:23:30.493683+0000","last_peered":"2024-04-25T01:23:30.493683+0000","last_clean":"2024-04-25T01:23:30.493683+0000","last_became_active":"2024-04-25T01:23:27.839038+0000","last_became_peered":"2024-04-25T01:23:27.839038+0000","last_unstale":"2024-04-25T01:23:30.493683+0000","last_undegraded":"2024-04-25T01:23:30.493683+0000","last_fullsized":"2024-04-25T01:23:30.493683+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:26.817415+0000","last_clean_scrub_stamp":"2024-04-25T01:23:26.817415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:28:34.927620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":226,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-25T01:23:30.492603+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:23:30.492603+0000","last_peered":"2024-04-25T01:23:30.492603+0000","last_clean":"2024-04-25T01:23:30.492603+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:23:30.492603+0000","last_undegraded":"2024-04-25T01:23:30.492603+0000","last_fullsized":"2024-04-25T01:23:30.492603+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1208320,"data_stored":860916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":614400,"data_stored":271076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":606208,"data_stored":270424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":610304,"data_stored":271006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":622592,"data_stored":272058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":272036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":614400,"data_stored":270584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28204,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343636,"statfs":{"total":96636764160,"available":96607883264,"internally_reserved":0,"allocated":1208320,"data_stored":860848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:23:36.476 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T01:23:36.476 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T01:23:36.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.635+0000 7f3ac74dd640 1 Processor -- start 2024-04-25T01:23:36.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.636+0000 7f3ac74dd640 1 -- start start 2024-04-25T01:23:36.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.636+0000 7f3ac74dd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ac015e4c0 0x7f3ac00d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.636+0000 7f3ac74dd640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ac00d1e90 con 0x7f3ac00d2d40 2024-04-25T01:23:36.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.636+0000 7f3ac74dd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ac00b54a0 con 0x7f3ac00d4a10 2024-04-25T01:23:36.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.636+0000 7f3ac74dd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3ac00d2130 con 0x7f3ac015e4c0 2024-04-25T01:23:36.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.636+0000 7f3ac5a7a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3ac00d2d40 0x7f3ac00d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48966/0 (socket says 172.21.15.77:48966) 2024-04-25T01:23:36.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.636+0000 7f3ac5a7a640 1 -- 172.21.15.77:0/4050988528 learned_addr learned my addr 172.21.15.77:0/4050988528 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:36.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.637+0000 7f3ac5279640 1 --2- 172.21.15.77:0/4050988528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ac015e4c0 0x7f3ac00d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.637+0000 7f3ab7fff640 1 -- 172.21.15.77:0/4050988528 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154863284 0 0) 0x7f3ac00d1e90 con 0x7f3ac00d2d40 2024-04-25T01:23:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.637+0000 7f3ab7fff640 1 -- 172.21.15.77:0/4050988528 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3aac003520 con 0x7f3ac00d2d40 2024-04-25T01:23:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.637+0000 7f3ab7fff640 1 -- 172.21.15.77:0/4050988528 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2670883558 0 0) 0x7f3ac00b54a0 con 0x7f3ac00d4a10 2024-04-25T01:23:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.637+0000 7f3ab7fff640 1 -- 172.21.15.77:0/4050988528 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ac00d1e90 con 0x7f3ac00d4a10 2024-04-25T01:23:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.637+0000 7f3ab7fff640 1 -- 172.21.15.77:0/4050988528 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2131185156 0 0) 0x7f3aac003520 con 0x7f3ac00d2d40 2024-04-25T01:23:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.637+0000 7f3ac5279640 1 -- 172.21.15.77:0/4050988528 >> v1:172.21.15.121:6789/0 conn(0x7f3ac00d4a10 legacy=0x7f3ac00b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.637+0000 7f3ac5279640 1 -- 172.21.15.77:0/4050988528 >> v1:172.21.15.77:6789/0 conn(0x7f3ac00d2d40 legacy=0x7f3ac00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.637+0000 7f3ac5279640 1 -- 172.21.15.77:0/4050988528 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac00b5780 con 0x7f3ac015e4c0 2024-04-25T01:23:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.638+0000 7f3ac5279640 1 --2- 172.21.15.77:0/4050988528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ac015e4c0 0x7f3ac00d2800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009dc0 tx=0x7f3abc02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41892d73a3ea87b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.638+0000 7f3ab7fff640 1 -- 172.21.15.77:0/4050988528 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8001670 con 0x7f3ac015e4c0 2024-04-25T01:23:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.638+0000 7f3ab7fff640 1 -- 172.21.15.77:0/4050988528 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3abc007540 con 0x7f3ac015e4c0 2024-04-25T01:23:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.638+0000 7f3ab7fff640 1 -- 172.21.15.77:0/4050988528 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc007910 con 0x7f3ac015e4c0 2024-04-25T01:23:36.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.638+0000 7f3ac74dd640 1 -- 172.21.15.77:0/4050988528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ac015e4c0 msgr2=0x7f3ac00d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.638+0000 7f3ac74dd640 1 --2- 172.21.15.77:0/4050988528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ac015e4c0 0x7f3ac00d2800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009dc0 tx=0x7f3abc02a5d0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.638+0000 7f3ac74dd640 1 -- 172.21.15.77:0/4050988528 shutdown_connections 2024-04-25T01:23:36.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.638+0000 7f3ac74dd640 1 --2- 172.21.15.77:0/4050988528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ac015e4c0 0x7f3ac00d2800 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.638+0000 7f3ac74dd640 1 -- 172.21.15.77:0/4050988528 >> 172.21.15.77:0/4050988528 conn(0x7f3ac00d3ef0 msgr2=0x7f3ac00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.639+0000 7f3ac74dd640 1 -- 172.21.15.77:0/4050988528 shutdown_connections 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.639+0000 7f3ac74dd640 1 -- 172.21.15.77:0/4050988528 wait complete. 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.640+0000 7f3ac74dd640 1 Processor -- start 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.640+0000 7f3ac74dd640 1 -- start start 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.640+0000 7f3ac74dd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac00d2d40 0x7f3ac00b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.640+0000 7f3ac74dd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ac00d4a10 0x7f3ac00b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.640+0000 7f3ac74dd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ac015e4c0 0x7f3ac01f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.640+0000 7f3ac74dd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac015ea20 con 0x7f3ac00d2d40 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.640+0000 7f3ac74dd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac015e8a0 con 0x7f3ac00d4a10 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.640+0000 7f3ac74dd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3ac00b45f0 con 0x7f3ac015e4c0 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.640+0000 7f3ac5279640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac00d2d40 0x7f3ac00b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.640+0000 7f3ac5279640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac00d2d40 0x7f3ac00b0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51380/0 (socket says 172.21.15.77:51380) 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.640+0000 7f3ac5279640 1 -- 172.21.15.77:0/1845636273 learned_addr learned my addr 172.21.15.77:0/1845636273 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:36.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ac5a7a640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ac015e4c0 0x7f3ac01f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ac4a78640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ac00d4a10 0x7f3ac00b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ac5279640 1 -- 172.21.15.77:0/1845636273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ac015e4c0 msgr2=0x7f3ac01f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ac5279640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ac015e4c0 0x7f3ac01f3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ac5279640 1 -- 172.21.15.77:0/1845636273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ac00d4a10 msgr2=0x7f3ac00b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ac5279640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ac00d4a10 0x7f3ac00b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ac5279640 1 -- 172.21.15.77:0/1845636273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac01f4ca0 con 0x7f3ac00d2d40 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ac5279640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac00d2d40 0x7f3ac00b0b80 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3abc000f80 tx=0x7f3abc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ab5ffb640 1 -- 172.21.15.77:0/1845636273 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc03f070 con 0x7f3ac00d2d40 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ab5ffb640 1 -- 172.21.15.77:0/1845636273 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3abc031c40 con 0x7f3ac00d2d40 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ab5ffb640 1 -- 172.21.15.77:0/1845636273 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc007e70 con 0x7f3ac00d2d40 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ac74dd640 1 -- 172.21.15.77:0/1845636273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac01f4f80 con 0x7f3ac00d2d40 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.641+0000 7f3ac74dd640 1 -- 172.21.15.77:0/1845636273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac01f5890 con 0x7f3ac00d2d40 2024-04-25T01:23:36.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.642+0000 7f3ac74dd640 1 -- 172.21.15.77:0/1845636273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac006d480 con 0x7f3ac00d2d40 2024-04-25T01:23:36.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.643+0000 7f3ab5ffb640 1 -- 172.21.15.77:0/1845636273 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3abc03b410 con 0x7f3ac00d2d40 2024-04-25T01:23:36.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.643+0000 7f3ab5ffb640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3aa406be30 0x7f3aa406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.643+0000 7f3ac4a78640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3aa406be30 0x7f3aa406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.643+0000 7f3ab5ffb640 1 -- 172.21.15.77:0/1845636273 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f3abc0ac750 con 0x7f3ac00d2d40 2024-04-25T01:23:36.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.644+0000 7f3ac4a78640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3aa406be30 0x7f3aa406e2f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3ac00b9ba0 tx=0x7f3ab0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:36.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.647+0000 7f3ab5ffb640 1 -- 172.21.15.77:0/1845636273 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3ac006d480 con 0x7f3ac00d2d40 2024-04-25T01:23:36.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.891+0000 7f3ac74dd640 1 -- 172.21.15.77:0/1845636273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f3ac0068fb0 con 0x7f3ac00d2d40 2024-04-25T01:23:36.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.893+0000 7f3ab5ffb640 1 -- 172.21.15.77:0/1845636273 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) ==== 190+0+147 (secure 0 0 0) 0x7f3abc07af70 con 0x7f3ac00d2d40 2024-04-25T01:23:36.875 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-04-25T01:23:36.875 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvrylmsfhcOhAAYZeVeMoyKmZdON0jUeFmeQ== 2024-04-25T01:23:36.875 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-04-25T01:23:36.875 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-04-25T01:23:36.875 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T01:23:36.875 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-04-25T01:23:36.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.895+0000 7f3ac74dd640 1 -- 172.21.15.77:0/1845636273 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3aa406be30 msgr2=0x7f3aa406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.895+0000 7f3ac74dd640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3aa406be30 0x7f3aa406e2f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3ac00b9ba0 tx=0x7f3ab0009040 comp rx=0 tx=0).stop 2024-04-25T01:23:36.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.895+0000 7f3ac74dd640 1 -- 172.21.15.77:0/1845636273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac00d2d40 msgr2=0x7f3ac00b0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.895+0000 7f3ac74dd640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac00d2d40 0x7f3ac00b0b80 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3abc000f80 tx=0x7f3abc004a40 comp rx=0 tx=0).stop 2024-04-25T01:23:36.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.896+0000 7f3ac74dd640 1 -- 172.21.15.77:0/1845636273 shutdown_connections 2024-04-25T01:23:36.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.896+0000 7f3ac74dd640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3aa406be30 0x7f3aa406e2f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.896+0000 7f3ac74dd640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ac015e4c0 0x7f3ac01f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.896+0000 7f3ac74dd640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ac00d4a10 0x7f3ac00b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.896+0000 7f3ac74dd640 1 --2- 172.21.15.77:0/1845636273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ac00d2d40 0x7f3ac00b0b80 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.896+0000 7f3ac74dd640 1 -- 172.21.15.77:0/1845636273 >> 172.21.15.77:0/1845636273 conn(0x7f3ac00d3ef0 msgr2=0x7f3ac015d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:36.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.896+0000 7f3ac74dd640 1 -- 172.21.15.77:0/1845636273 shutdown_connections 2024-04-25T01:23:36.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:36.896+0000 7f3ac74dd640 1 -- 172.21.15.77:0/1845636273 wait complete. 2024-04-25T01:23:36.887 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:36.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.002+0000 7f9e500d2640 1 Processor -- start 2024-04-25T01:23:36.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.003+0000 7f9e500d2640 1 -- start start 2024-04-25T01:23:36.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.003+0000 7f9e500d2640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9e4815e4c0 0x7f9e480d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.003+0000 7f9e500d2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e480d4a10 0x7f9e480b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.003+0000 7f9e500d2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e480d1e90 con 0x7f9e480d2d40 2024-04-25T01:23:36.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.003+0000 7f9e500d2640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9e480b54a0 con 0x7f9e4815e4c0 2024-04-25T01:23:36.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.003+0000 7f9e500d2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9e480d2130 con 0x7f9e480d4a10 2024-04-25T01:23:36.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.004+0000 7f9e4d66d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e480d4a10 0x7f9e480b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.004+0000 7f9e4d66d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e480d4a10 0x7f9e480b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:41870/0 (socket says 172.21.15.77:41870) 2024-04-25T01:23:36.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.004+0000 7f9e4d66d640 1 -- 172.21.15.77:0/2123129524 learned_addr learned my addr 172.21.15.77:0/2123129524 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:36.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.004+0000 7f9e4de6e640 1 --2- 172.21.15.77:0/2123129524 >> v2:172.21.15.121:3300/0 conn(0x7f9e4815e4c0 0x7f9e480d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.004+0000 7f9e4de6e640 1 -- 172.21.15.77:0/2123129524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e480d4a10 msgr2=0x7f9e480b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:36.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.004+0000 7f9e4de6e640 1 --2- 172.21.15.77:0/2123129524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e480d4a10 0x7f9e480b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.004+0000 7f9e4de6e640 1 -- 172.21.15.77:0/2123129524 >> v1:172.21.15.121:6789/0 conn(0x7f9e480d2d40 legacy=0x7f9e480d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.004+0000 7f9e4de6e640 1 -- 172.21.15.77:0/2123129524 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e480b5780 con 0x7f9e4815e4c0 2024-04-25T01:23:36.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.004+0000 7f9e4d66d640 1 --2- 172.21.15.77:0/2123129524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e480d4a10 0x7f9e480b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:36.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.005+0000 7f9e4de6e640 1 --2- 172.21.15.77:0/2123129524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e4815e4c0 0x7f9e480d2800 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c0097a0 tx=0x7f9e3c02a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bf30eba54ce0c82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:36.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.005+0000 7f9e4ce6c640 1 -- 172.21.15.77:0/2123129524 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c037070 con 0x7f9e4815e4c0 2024-04-25T01:23:36.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.005+0000 7f9e4ce6c640 1 -- 172.21.15.77:0/2123129524 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e3c02ae50 con 0x7f9e4815e4c0 2024-04-25T01:23:36.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.005+0000 7f9e4ce6c640 1 -- 172.21.15.77:0/2123129524 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c032660 con 0x7f9e4815e4c0 2024-04-25T01:23:36.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.005+0000 7f9e500d2640 1 -- 172.21.15.77:0/2123129524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e4815e4c0 msgr2=0x7f9e480d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.006+0000 7f9e500d2640 1 --2- 172.21.15.77:0/2123129524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e4815e4c0 0x7f9e480d2800 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c0097a0 tx=0x7f9e3c02a100 comp rx=0 tx=0).stop 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.006+0000 7f9e500d2640 1 -- 172.21.15.77:0/2123129524 shutdown_connections 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.006+0000 7f9e500d2640 1 --2- 172.21.15.77:0/2123129524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e480d4a10 0x7f9e480b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.006+0000 7f9e500d2640 1 --2- 172.21.15.77:0/2123129524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e4815e4c0 0x7f9e480d2800 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.006+0000 7f9e500d2640 1 -- 172.21.15.77:0/2123129524 >> 172.21.15.77:0/2123129524 conn(0x7f9e480d3ef0 msgr2=0x7f9e480d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.006+0000 7f9e500d2640 1 -- 172.21.15.77:0/2123129524 shutdown_connections 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.006+0000 7f9e500d2640 1 -- 172.21.15.77:0/2123129524 wait complete. 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.007+0000 7f9e500d2640 1 Processor -- start 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.007+0000 7f9e500d2640 1 -- start start 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.007+0000 7f9e500d2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e480d2d40 0x7f9e48151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.007+0000 7f9e500d2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e480d4a10 0x7f9e480b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.007+0000 7f9e500d2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e4815e4c0 0x7f9e480b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.007+0000 7f9e500d2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4815ea20 con 0x7f9e480d2d40 2024-04-25T01:23:36.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.007+0000 7f9e500d2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4815e8a0 con 0x7f9e480d4a10 2024-04-25T01:23:36.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.007+0000 7f9e500d2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9e480d1010 con 0x7f9e4815e4c0 2024-04-25T01:23:36.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4de6e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e480d2d40 0x7f9e48151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4de6e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e480d2d40 0x7f9e48151f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51392/0 (socket says 172.21.15.77:51392) 2024-04-25T01:23:36.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4de6e640 1 -- 172.21.15.77:0/3767910989 learned_addr learned my addr 172.21.15.77:0/3767910989 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:36.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4e66f640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e4815e4c0 0x7f9e480b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4d66d640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e480d4a10 0x7f9e480b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4de6e640 1 -- 172.21.15.77:0/3767910989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e4815e4c0 msgr2=0x7f9e480b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4de6e640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e4815e4c0 0x7f9e480b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4de6e640 1 -- 172.21.15.77:0/3767910989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e480d4a10 msgr2=0x7f9e480b0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:36.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4de6e640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e480d4a10 0x7f9e480b0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:36.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4de6e640 1 -- 172.21.15.77:0/3767910989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e480b1520 con 0x7f9e480d2d40 2024-04-25T01:23:36.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4d66d640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e480d4a10 0x7f9e480b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:36.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e4de6e640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e480d2d40 0x7f9e48151f20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c0098c0 tx=0x7f9e3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:36.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e3a7fc640 1 -- 172.21.15.77:0/3767910989 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c042070 con 0x7f9e480d2d40 2024-04-25T01:23:36.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e3a7fc640 1 -- 172.21.15.77:0/3767910989 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e3c03e460 con 0x7f9e480d2d40 2024-04-25T01:23:36.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e500d2640 1 -- 172.21.15.77:0/3767910989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e480b1800 con 0x7f9e480d2d40 2024-04-25T01:23:36.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e3a7fc640 1 -- 172.21.15.77:0/3767910989 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c037040 con 0x7f9e480d2d40 2024-04-25T01:23:36.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.008+0000 7f9e500d2640 1 -- 172.21.15.77:0/3767910989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e481f4ae0 con 0x7f9e480d2d40 2024-04-25T01:23:36.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.009+0000 7f9e500d2640 1 -- 172.21.15.77:0/3767910989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e48068fb0 con 0x7f9e480d2d40 2024-04-25T01:23:36.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.010+0000 7f9e3a7fc640 1 -- 172.21.15.77:0/3767910989 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9e3c045070 con 0x7f9e480d2d40 2024-04-25T01:23:36.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.010+0000 7f9e3a7fc640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e2c06be30 0x7f9e2c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:36.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.010+0000 7f9e3a7fc640 1 -- 172.21.15.77:0/3767910989 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f9e3c0ab6a0 con 0x7f9e480d2d40 2024-04-25T01:23:36.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.010+0000 7f9e4d66d640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e2c06be30 0x7f9e2c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:36.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.010+0000 7f9e4d66d640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e2c06be30 0x7f9e2c06e2f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9e480b1b00 tx=0x7f9e44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:36.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.014+0000 7f9e3a7fc640 1 -- 172.21.15.77:0/3767910989 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9e3c0797f0 con 0x7f9e480d2d40 2024-04-25T01:23:37.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.224+0000 7f9e500d2640 1 -- 172.21.15.77:0/3767910989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e480cad10 con 0x7f9e480d2d40 2024-04-25T01:23:37.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.224+0000 7f9e3a7fc640 1 -- 172.21.15.77:0/3767910989 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5184 (secure 0 0 0) 0x7f9e3c07daf0 con 0x7f9e480d2d40 2024-04-25T01:23:37.205 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:37.206 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5254,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/865504322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":865504322},{"type":"v1","addr":"172.21.15.121:6835","nonce":865504322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":5886,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3189776231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3189776231},{"type":"v1","addr":"172.21.15.77:6836","nonce":3189776231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3454287131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3454287131},{"type":"v1","addr":"172.21.15.77:6837","nonce":3454287131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:23:28.170422+0000","modified":"2024-04-25T01:23:30.191645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6834/3834135103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3834135103},{"type":"v1","addr":"172.21.15.121:6834","nonce":3834135103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":3}]} 2024-04-25T01:23:37.206 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 27 2024-04-25T01:23:37.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 -- 172.21.15.77:0/3767910989 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e2c06be30 msgr2=0x7f9e2c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e2c06be30 0x7f9e2c06e2f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9e480b1b00 tx=0x7f9e44008040 comp rx=0 tx=0).stop 2024-04-25T01:23:37.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 -- 172.21.15.77:0/3767910989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e480d2d40 msgr2=0x7f9e48151f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e480d2d40 0x7f9e48151f20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c0098c0 tx=0x7f9e3c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 -- 172.21.15.77:0/3767910989 shutdown_connections 2024-04-25T01:23:37.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9e2c06be30 0x7f9e2c06e2f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e4815e4c0 0x7f9e480b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e480d4a10 0x7f9e480b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 --2- 172.21.15.77:0/3767910989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e480d2d40 0x7f9e48151f20 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 -- 172.21.15.77:0/3767910989 >> 172.21.15.77:0/3767910989 conn(0x7f9e480d3ef0 msgr2=0x7f9e480b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:37.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 -- 172.21.15.77:0/3767910989 shutdown_connections 2024-04-25T01:23:37.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.226+0000 7f9e500d2640 1 -- 172.21.15.77:0/3767910989 wait complete. 2024-04-25T01:23:37.620 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:23:28.170422+0000', 'modified': '2024-04-25T01:23:30.191645+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5901}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5901': {'gid': 5901, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6834/3834135103', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6832', 'nonce': 3834135103}, {'type': 'v1', 'addr': '172.21.15.121:6834', 'nonce': 3834135103}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5901, 'qdb_cluster': [5901]} 2024-04-25T01:23:37.620 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:23:37.621 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:23:37.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.741+0000 7fe3bacc3640 1 Processor -- start 2024-04-25T01:23:37.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.741+0000 7fe3bacc3640 1 -- start start 2024-04-25T01:23:37.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3bacc3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe3b4143940 0x7fe3b406e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3bacc3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3b4143d20 0x7fe3b406eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3bacc3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3b4070e60 con 0x7fe3b415ecf0 2024-04-25T01:23:37.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3bacc3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe3b4071140 con 0x7fe3b4143940 2024-04-25T01:23:37.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3bacc3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe3b4071420 con 0x7fe3b4143d20 2024-04-25T01:23:37.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3b8a5f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe3b4143940 0x7fe3b406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3b3fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3b4143d20 0x7fe3b406eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3b9260640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe3b415ecf0 0x7fe3b415f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:54858/0 (socket says 172.21.15.77:54858) 2024-04-25T01:23:37.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3b9260640 1 -- 172.21.15.77:0/911246438 learned_addr learned my addr 172.21.15.77:0/911246438 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:23:37.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3b3fff640 1 -- 172.21.15.77:0/911246438 >> v2:172.21.15.121:3300/0 conn(0x7fe3b4143940 msgr2=0x7fe3b406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:37.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/911246438 >> v2:172.21.15.121:3300/0 conn(0x7fe3b4143940 0x7fe3b406e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3b3fff640 1 -- 172.21.15.77:0/911246438 >> v1:172.21.15.121:6789/0 conn(0x7fe3b415ecf0 legacy=0x7fe3b415f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.742+0000 7fe3b3fff640 1 -- 172.21.15.77:0/911246438 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b4071700 con 0x7fe3b4143d20 2024-04-25T01:23:37.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.743+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/911246438 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3b4143d20 0x7fe3b406eb90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00097a0 tx=0x7fe3a002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73486ff8e43e8779 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.743+0000 7fe3b37fe640 1 -- 172.21.15.77:0/911246438 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0037070 con 0x7fe3b4143d20 2024-04-25T01:23:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.743+0000 7fe3b37fe640 1 -- 172.21.15.77:0/911246438 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3a0032440 con 0x7fe3b4143d20 2024-04-25T01:23:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.743+0000 7fe3b37fe640 1 -- 172.21.15.77:0/911246438 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0032810 con 0x7fe3b4143d20 2024-04-25T01:23:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.744+0000 7fe3bacc3640 1 -- 172.21.15.77:0/911246438 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3b4143d20 msgr2=0x7fe3b406eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.744+0000 7fe3bacc3640 1 --2- 172.21.15.77:0/911246438 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3b4143d20 0x7fe3b406eb90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00097a0 tx=0x7fe3a002a180 comp rx=0 tx=0).stop 2024-04-25T01:23:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.744+0000 7fe3bacc3640 1 -- 172.21.15.77:0/911246438 shutdown_connections 2024-04-25T01:23:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.744+0000 7fe3bacc3640 1 --2- 172.21.15.77:0/911246438 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3b4143d20 0x7fe3b406eb90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.744+0000 7fe3bacc3640 1 --2- 172.21.15.77:0/911246438 >> v2:172.21.15.121:3300/0 conn(0x7fe3b4143940 0x7fe3b406e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.744+0000 7fe3bacc3640 1 -- 172.21.15.77:0/911246438 >> 172.21.15.77:0/911246438 conn(0x7fe3b40b1820 msgr2=0x7fe3b40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.744+0000 7fe3bacc3640 1 -- 172.21.15.77:0/911246438 shutdown_connections 2024-04-25T01:23:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.744+0000 7fe3bacc3640 1 -- 172.21.15.77:0/911246438 wait complete. 2024-04-25T01:23:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.745+0000 7fe3bacc3640 1 Processor -- start 2024-04-25T01:23:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.745+0000 7fe3bacc3640 1 -- start start 2024-04-25T01:23:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3bacc3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3b4143940 0x7fe3b413f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3bacc3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3b4143d20 0x7fe3b4154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3bacc3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3b415ecf0 0x7fe3b40b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3bacc3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b406d8d0 con 0x7fe3b4143940 2024-04-25T01:23:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3bacc3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b406d750 con 0x7fe3b4143d20 2024-04-25T01:23:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3bacc3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe3b406da50 con 0x7fe3b415ecf0 2024-04-25T01:23:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3b8a5f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3b4143940 0x7fe3b413f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3b8a5f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3b4143940 0x7fe3b413f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51396/0 (socket says 172.21.15.77:51396) 2024-04-25T01:23:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3b8a5f640 1 -- 172.21.15.77:0/1349609832 learned_addr learned my addr 172.21.15.77:0/1349609832 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3b4143d20 0x7fe3b4154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3b9260640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3b415ecf0 0x7fe3b40b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3b8a5f640 1 -- 172.21.15.77:0/1349609832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3b415ecf0 msgr2=0x7fe3b40b0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3b415ecf0 0x7fe3b40b0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.746+0000 7fe3b8a5f640 1 -- 172.21.15.77:0/1349609832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3b4143d20 msgr2=0x7fe3b4154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.747+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3b4143d20 0x7fe3b4154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.747+0000 7fe3b8a5f640 1 -- 172.21.15.77:0/1349609832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b4156a40 con 0x7fe3b4143940 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.747+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3b4143d20 0x7fe3b4154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.747+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3b4143940 0x7fe3b413f390 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800e9b0 tx=0x7fe3a800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.747+0000 7fe3b17fa640 1 -- 172.21.15.77:0/1349609832 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8017070 con 0x7fe3b4143940 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.747+0000 7fe3b17fa640 1 -- 172.21.15.77:0/1349609832 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3a800c9a0 con 0x7fe3b4143940 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.747+0000 7fe3bacc3640 1 -- 172.21.15.77:0/1349609832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3b4165f80 con 0x7fe3b4143940 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.747+0000 7fe3b17fa640 1 -- 172.21.15.77:0/1349609832 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a800cd90 con 0x7fe3b4143940 2024-04-25T01:23:37.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.747+0000 7fe3bacc3640 1 -- 172.21.15.77:0/1349609832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3b4166540 con 0x7fe3b4143940 2024-04-25T01:23:37.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.748+0000 7fe3b17fa640 1 -- 172.21.15.77:0/1349609832 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe3a801a070 con 0x7fe3b4143940 2024-04-25T01:23:37.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.749+0000 7fe3b17fa640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe39806be30 0x7fe39806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.749+0000 7fe3b17fa640 1 -- 172.21.15.77:0/1349609832 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fe3a808afe0 con 0x7fe3b4143940 2024-04-25T01:23:37.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.749+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe39806be30 0x7fe39806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.749+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe39806be30 0x7fe39806e2f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0034000 tx=0x7fe3a002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3bd9cc740 1 Processor -- start 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3bd9cc740 1 -- start start 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f43db60 0x56539f3f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f420c80 0x56539f420000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420540 0x56539f3f5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56539f358f00 con 0x56539f420c80 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56539f301750 con 0x56539f420540 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56539f354ec0 con 0x56539f43db60 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3b3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f420c80 0x56539f420000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3b3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f420c80 0x56539f420000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51412/0 (socket says 172.21.15.77:51412) 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3b3fff640 1 -- 172.21.15.77:0/978629722 learned_addr learned my addr 172.21.15.77:0/978629722 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/978629722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f43db60 0x56539f3f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.750+0000 7fe3b9260640 1 --2- 172.21.15.77:0/978629722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420540 0x56539f3f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe3b3fff640 1 -- 172.21.15.77:0/978629722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f43db60 msgr2=0x56539f3f04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/978629722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f43db60 0x56539f3f04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe3b3fff640 1 -- 172.21.15.77:0/978629722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420540 msgr2=0x56539f3f5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/978629722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420540 0x56539f3f5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe3b3fff640 1 -- 172.21.15.77:0/978629722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56539f06d2d0 con 0x56539f420c80 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/978629722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f43db60 0x56539f3f04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe3b9260640 1 --2- 172.21.15.77:0/978629722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420540 0x56539f3f5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/978629722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f420c80 0x56539f420000 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0032810 tx=0x7fe3a002c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b16ac682ec91636 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe39e7fc640 1 -- 172.21.15.77:0/978629722 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a004d070 con 0x56539f420c80 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe39e7fc640 1 -- 172.21.15.77:0/978629722 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3a003e460 con 0x56539f420c80 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe39e7fc640 1 -- 172.21.15.77:0/978629722 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0037040 con 0x56539f420c80 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/978629722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f420c80 msgr2=0x56539f420000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.751+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/978629722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f420c80 0x56539f420000 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0032810 tx=0x7fe3a002c000 comp rx=0 tx=0).stop 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.752+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/978629722 shutdown_connections 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.752+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/978629722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f420c80 0x56539f420000 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.752+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/978629722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420540 0x56539f3f5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.752+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/978629722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f43db60 0x56539f3f04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.752+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/978629722 >> 172.21.15.77:0/978629722 conn(0x56539f3f2760 msgr2=0x56539f43b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.752+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/978629722 shutdown_connections 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.752+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/978629722 wait complete. 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.752+0000 7fe3bd9cc740 1 -- start start 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.752+0000 7fe3bd9cc740 1 Processor -- start 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420c80 0x56539f41b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f43db60 0x56539f43afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f43b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56539f3f4910 con 0x56539f437ef0 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56539f314cf0 con 0x56539f420c80 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56539f3f3d00 con 0x56539f43db60 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3bd9cc740 10 client.0 _ll_register_callbacks cb 0x56539f3146e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:23:37.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3b9260640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f43b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3b9260640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f43b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51426/0 (socket says 172.21.15.77:51426) 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3b9260640 1 -- 172.21.15.77:0/144667312 learned_addr learned my addr 172.21.15.77:0/144667312 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420c80 0x56539f41b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f43db60 0x56539f43afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3b9260640 1 -- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f43db60 msgr2=0x56539f43afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3b9260640 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f43db60 0x56539f43afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3b9260640 1 -- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420c80 msgr2=0x56539f41b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3b9260640 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420c80 0x56539f41b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.753+0000 7fe3b9260640 1 -- 172.21.15.77:0/144667312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56539f1376b0 con 0x56539f437ef0 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420c80 0x56539f41b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3b9260640 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f43b4f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac007500 tx=0x7fe3ac0075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42583d9fd50bb941 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3937fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3937fe640 1 -- 172.21.15.77:0/144667312 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac018030 con 0x56539f437ef0 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3937fe640 1 -- 172.21.15.77:0/144667312 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fe3ac014610 con 0x56539f437ef0 2024-04-25T01:23:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/144667312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3a003e460 con 0x56539f437ef0 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3937fe640 1 -- 172.21.15.77:0/144667312 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe3ac021050 con 0x56539f437ef0 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3bd9cc740 10 client.6048 fetch_fsmap learned FSMap version 27 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/144667312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) -- 0x56539f44b190 con 0x56539f437ef0 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3937fe640 1 -- 172.21.15.77:0/144667312 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3a003e460 con 0x56539f437ef0 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3937fe640 1 -- 172.21.15.77:0/144667312 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 3494+0+0 (secure 0 0 0) 0x7fe3ac014810 con 0x56539f437ef0 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3bd9cc740 10 client.6048 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3bd9cc740 10 client.6048 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3bd9cc740 20 client.6048 populate_metadata read hostname 'smithi077' 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:6832/3834135103,v1:172.21.15.121:6834/3834135103] conn(0x56539f422910 0x56539f4217b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3bd9cc740 4 client.6048 mds_command: new command op to 5901 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.754+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/144667312 --> [v2:172.21.15.121:6832/3834135103,v1:172.21.15.121:6834/3834135103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56539f366440 con 0x56539f422910 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.755+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:6832/3834135103,v1:172.21.15.121:6834/3834135103] conn(0x56539f422910 0x56539f4217b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.755+0000 7fe3937fe640 10 client.6048 ms_handle_connect on v2:172.21.15.121:6832/3834135103 2024-04-25T01:23:37.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.755+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:6832/3834135103,v1:172.21.15.121:6834/3834135103] conn(0x56539f422910 0x56539f4217b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaa1280d4a16f6f8 server_cookie=518e60d0eeb449ab in_seq=0 out_seq=0 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3937fe640 1 -- 172.21.15.77:0/144667312 <== mds.0 v2:172.21.15.121:6832/3834135103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56539f366440 con 0x56539f422910 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3937fe640 10 client.6048 handle_command_reply: tid=0 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 1 client.6048 shutdown 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/144667312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 msgr2=0x56539f43b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f43b4f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac007500 tx=0x7fe3ac0075f0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/144667312 shutdown_connections 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:6832/3834135103,v1:172.21.15.121:6834/3834135103] conn(0x56539f422910 0x56539f4217b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f43b4f0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f43db60 0x56539f43afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3937fe640 0 client.6048 ms_handle_reset on v2:172.21.15.121:6832/3834135103 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/144667312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420c80 0x56539f41b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/144667312 >> 172.21.15.77:0/144667312 conn(0x56539f420540 msgr2=0x56539f43c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/144667312 shutdown_connections 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/144667312 wait complete. 2024-04-25T01:23:37.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.756+0000 7fe3bd9cc740 20 client.6048 trim_cache size 0 max 16384 2024-04-25T01:23:37.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.774+0000 7fe3bd9cc740 1 Processor -- start 2024-04-25T01:23:37.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.774+0000 7fe3bd9cc740 1 -- start start 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.774+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f420c80 0x56539f4324d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.774+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f44dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.774+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f43db60 0x56539f4acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.774+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b41ffd00 con 0x56539f437ef0 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.774+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b41f8520 con 0x56539f43db60 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.774+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe3b4157b90 con 0x56539f420c80 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe3b3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f44dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe3b3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f44dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51442/0 (socket says 172.21.15.77:51442) 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe3b3fff640 1 -- 172.21.15.77:0/775708603 learned_addr learned my addr 172.21.15.77:0/775708603 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe3b3fff640 1 -- 172.21.15.77:0/775708603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f420c80 msgr2=0x56539f4324d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/775708603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f420c80 0x56539f4324d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe3b3fff640 1 -- 172.21.15.77:0/775708603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f43db60 msgr2=0x56539f4acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/775708603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f43db60 0x56539f4acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe3b3fff640 1 -- 172.21.15.77:0/775708603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56539f3f18b0 con 0x56539f437ef0 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/775708603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f44dad0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0000c00 tx=0x7fe3a003b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ff5ebde14efe39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe39dffb640 1 -- 172.21.15.77:0/775708603 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a004d070 con 0x56539f437ef0 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe39dffb640 1 -- 172.21.15.77:0/775708603 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3a003e6f0 con 0x56539f437ef0 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.775+0000 7fe39dffb640 1 -- 172.21.15.77:0/775708603 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0037040 con 0x56539f437ef0 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.776+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/775708603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 msgr2=0x56539f44dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.776+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/775708603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f44dad0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0000c00 tx=0x7fe3a003b000 comp rx=0 tx=0).stop 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.776+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/775708603 shutdown_connections 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.776+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/775708603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f437ef0 0x56539f44dad0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.776+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/775708603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f43db60 0x56539f4acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.776+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/775708603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f420c80 0x56539f4324d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.776+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/775708603 >> 172.21.15.77:0/775708603 conn(0x56539f420540 msgr2=0x56539f3eb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.776+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/775708603 shutdown_connections 2024-04-25T01:23:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.776+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/775708603 wait complete. 2024-04-25T01:23:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.776+0000 7fe3bd9cc740 1 -- start start 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.776+0000 7fe3bd9cc740 1 Processor -- start 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420c80 0x56539f49adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f437ef0 0x56539f419ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3bd9cc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f43db60 0x56539f41a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56539f422790 con 0x56539f43db60 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56539f439d00 con 0x56539f420c80 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3bd9cc740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56539f422de0 con 0x56539f437ef0 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3bd9cc740 10 client.0 _ll_register_callbacks cb 0x7fe3ac0070e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3b9260640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f43db60 0x56539f41a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3b9260640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f43db60 0x56539f41a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51456/0 (socket says 172.21.15.77:51456) 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3b9260640 1 -- 172.21.15.77:0/1310044194 learned_addr learned my addr 172.21.15.77:0/1310044194 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420c80 0x56539f49adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.777+0000 7fe3b3fff640 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f437ef0 0x56539f419ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3b9260640 1 -- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f437ef0 msgr2=0x56539f419ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3b9260640 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f437ef0 0x56539f419ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3b9260640 1 -- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420c80 msgr2=0x56539f49adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3b9260640 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420c80 0x56539f49adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3b9260640 1 -- 172.21.15.77:0/1310044194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56539f4ad2b0 con 0x56539f43db60 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3b9260640 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f43db60 0x56539f41a420 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac007420 tx=0x7fe3ac001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3bb882b4ae61291 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3937fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3937fe640 1 -- 172.21.15.77:0/1310044194 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac018070 con 0x56539f43db60 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3937fe640 1 -- 172.21.15.77:0/1310044194 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fe3ac014820 con 0x56539f43db60 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/1310044194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3a003e6f0 con 0x56539f43db60 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3937fe640 1 -- 172.21.15.77:0/1310044194 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe3ac01d420 con 0x56539f43db60 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3bd9cc740 10 client.6054 fetch_fsmap learned FSMap version 27 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/1310044194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) -- 0x56539f41aad0 con 0x56539f43db60 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3937fe640 1 -- 172.21.15.77:0/1310044194 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3a003e6f0 con 0x56539f43db60 2024-04-25T01:23:37.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3937fe640 1 -- 172.21.15.77:0/1310044194 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 3494+0+0 (secure 0 0 0) 0x7fe3ac01d5c0 con 0x56539f43db60 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3bd9cc740 10 client.6054 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3bd9cc740 10 client.6054 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3bd9cc740 20 client.6054 populate_metadata read hostname 'smithi077' 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:6832/3834135103,v1:172.21.15.121:6834/3834135103] conn(0x56539f430fe0 0x56539f43e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3bd9cc740 4 client.6054 mds_command: new command op to 5901 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/1310044194 --> [v2:172.21.15.121:6832/3834135103,v1:172.21.15.121:6834/3834135103] -- command(tid 0: {"prefix": "status"}) -- 0x56539f360790 con 0x56539f430fe0 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.778+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:6832/3834135103,v1:172.21.15.121:6834/3834135103] conn(0x56539f430fe0 0x56539f43e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3937fe640 10 client.6054 ms_handle_connect on v2:172.21.15.121:6832/3834135103 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3b8a5f640 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:6832/3834135103,v1:172.21.15.121:6834/3834135103] conn(0x56539f430fe0 0x56539f43e420 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbc82b3935fdb8ee server_cookie=a3cbf32036605c74 in_seq=0 out_seq=0 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3937fe640 1 -- 172.21.15.77:0/1310044194 <== mds.0 v2:172.21.15.121:6832/3834135103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x56539f360790 con 0x56539f430fe0 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3937fe640 10 client.6054 handle_command_reply: tid=0 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 1 client.6054 shutdown 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/1310044194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f43db60 msgr2=0x56539f41a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f43db60 0x56539f41a420 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac007420 tx=0x7fe3ac001250 comp rx=0 tx=0).stop 2024-04-25T01:23:37.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/1310044194 shutdown_connections 2024-04-25T01:23:37.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:6832/3834135103,v1:172.21.15.121:6834/3834135103] conn(0x56539f430fe0 0x56539f43e420 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56539f43db60 0x56539f41a420 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56539f437ef0 0x56539f419ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3937fe640 0 client.6054 ms_handle_reset on v2:172.21.15.121:6832/3834135103 2024-04-25T01:23:37.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 1 --2- 172.21.15.77:0/1310044194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539f420c80 0x56539f49adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/1310044194 >> 172.21.15.77:0/1310044194 conn(0x56539f420540 msgr2=0x56539f44db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:37.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/1310044194 shutdown_connections 2024-04-25T01:23:37.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 1 -- 172.21.15.77:0/1310044194 wait complete. 2024-04-25T01:23:37.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.779+0000 7fe3bd9cc740 20 client.6054 trim_cache size 0 max 16384 2024-04-25T01:23:37.766 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "7c3c39ee-077e-476c-8994-eede4fab301d", 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 5901, 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 8.9348080190000001, 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 27, 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 51, 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 51, 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 13.996737333 2024-04-25T01:23:37.767 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:23:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 -- 172.21.15.77:0/1349609832 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe39806be30 msgr2=0x7fe39806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe39806be30 0x7fe39806e2f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0034000 tx=0x7fe3a002e000 comp rx=0 tx=0).stop 2024-04-25T01:23:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 -- 172.21.15.77:0/1349609832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3b4143940 msgr2=0x7fe3b413f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3b4143940 0x7fe3b413f390 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800e9b0 tx=0x7fe3a800ee70 comp rx=0 tx=0).stop 2024-04-25T01:23:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 -- 172.21.15.77:0/1349609832 shutdown_connections 2024-04-25T01:23:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe39806be30 0x7fe39806e2f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3b415ecf0 0x7fe3b40b0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3b4143d20 0x7fe3b4154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 --2- 172.21.15.77:0/1349609832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3b4143940 0x7fe3b413f390 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 -- 172.21.15.77:0/1349609832 >> 172.21.15.77:0/1349609832 conn(0x7fe3b40b1820 msgr2=0x7fe3b415f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 -- 172.21.15.77:0/1349609832 shutdown_connections 2024-04-25T01:23:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.780+0000 7fe3bacc3640 1 -- 172.21.15.77:0/1349609832 wait complete. 2024-04-25T01:23:37.772 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:23:37.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.887+0000 7f87dddc8640 1 Processor -- start 2024-04-25T01:23:37.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.887+0000 7f87dddc8640 1 -- start start 2024-04-25T01:23:37.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87dddc8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87d806fcd0 0x7f87d815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87dddc8640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f87d80ade10 0x7f87d806c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87dddc8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d806c670 0x7f87d8161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87dddc8640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f87d81589e0 con 0x7f87d806fcd0 2024-04-25T01:23:37.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87dddc8640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f87d80b3890 con 0x7f87d80ade10 2024-04-25T01:23:37.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87dddc8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f87d80ae2c0 con 0x7f87d806c670 2024-04-25T01:23:37.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87d77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87d806fcd0 0x7f87d815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87d77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87d806fcd0 0x7f87d815ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51470/0 (socket says 172.21.15.77:51470) 2024-04-25T01:23:37.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87d77fe640 1 -- 172.21.15.77:0/503683873 learned_addr learned my addr 172.21.15.77:0/503683873 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:37.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87d77fe640 1 -- 172.21.15.77:0/503683873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d806c670 msgr2=0x7f87d8161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:37.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87d77fe640 1 --2- 172.21.15.77:0/503683873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d806c670 0x7f87d8161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87d77fe640 1 -- 172.21.15.77:0/503683873 >> v2:172.21.15.121:3300/0 conn(0x7f87d80ade10 msgr2=0x7f87d806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:37.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87d77fe640 1 --2- 172.21.15.77:0/503683873 >> v2:172.21.15.121:3300/0 conn(0x7f87d80ade10 0x7f87d806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.888+0000 7f87d77fe640 1 -- 172.21.15.77:0/503683873 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d80b3560 con 0x7f87d806fcd0 2024-04-25T01:23:37.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.889+0000 7f87d77fe640 1 --2- 172.21.15.77:0/503683873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d806fcd0 0x7f87d815ef50 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f87d80724a0 tx=0x7f87c802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef1b7c9c2ef03fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.889+0000 7f87d67fc640 1 -- 172.21.15.77:0/503683873 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8037070 con 0x7f87d806fcd0 2024-04-25T01:23:37.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.889+0000 7f87d67fc640 1 -- 172.21.15.77:0/503683873 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c8032440 con 0x7f87d806fcd0 2024-04-25T01:23:37.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.889+0000 7f87d67fc640 1 -- 172.21.15.77:0/503683873 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8032810 con 0x7f87d806fcd0 2024-04-25T01:23:37.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.890+0000 7f87dddc8640 1 -- 172.21.15.77:0/503683873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d806fcd0 msgr2=0x7f87d815ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.890+0000 7f87dddc8640 1 --2- 172.21.15.77:0/503683873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d806fcd0 0x7f87d815ef50 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f87d80724a0 tx=0x7f87c802a140 comp rx=0 tx=0).stop 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.890+0000 7f87dddc8640 1 -- 172.21.15.77:0/503683873 shutdown_connections 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.890+0000 7f87dddc8640 1 --2- 172.21.15.77:0/503683873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d806c670 0x7f87d8161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.890+0000 7f87dddc8640 1 --2- 172.21.15.77:0/503683873 >> v2:172.21.15.121:3300/0 conn(0x7f87d80ade10 0x7f87d806c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.890+0000 7f87dddc8640 1 --2- 172.21.15.77:0/503683873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d806fcd0 0x7f87d815ef50 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.890+0000 7f87dddc8640 1 -- 172.21.15.77:0/503683873 >> 172.21.15.77:0/503683873 conn(0x7f87d8071550 msgr2=0x7f87d8071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.890+0000 7f87dddc8640 1 -- 172.21.15.77:0/503683873 shutdown_connections 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.890+0000 7f87dddc8640 1 -- 172.21.15.77:0/503683873 wait complete. 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.891+0000 7f87dddc8640 1 Processor -- start 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.891+0000 7f87dddc8640 1 -- start start 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87dddc8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d806c670 0x7f87d80ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87dddc8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d806fcd0 0x7f87d814ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87dddc8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d80ade10 0x7f87d814f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87dddc8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f87d80ae7f0 con 0x7f87d806fcd0 2024-04-25T01:23:37.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87dddc8640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f87d806ca90 con 0x7f87d806c670 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87dddc8640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f87d80aead0 con 0x7f87d80ade10 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d806fcd0 0x7f87d814ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d806fcd0 0x7f87d814ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51476/0 (socket says 172.21.15.77:51476) 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d6ffd640 1 -- 172.21.15.77:0/3503397422 learned_addr learned my addr 172.21.15.77:0/3503397422 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d77fe640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d806c670 0x7f87d80ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d7fff640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d80ade10 0x7f87d814f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d6ffd640 1 -- 172.21.15.77:0/3503397422 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d80ade10 msgr2=0x7f87d814f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d6ffd640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d80ade10 0x7f87d814f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d6ffd640 1 -- 172.21.15.77:0/3503397422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d806c670 msgr2=0x7f87d80ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d6ffd640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d806c670 0x7f87d80ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d6ffd640 1 -- 172.21.15.77:0/3503397422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d814d0a0 con 0x7f87d806fcd0 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d77fe640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d806c670 0x7f87d80ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87d6ffd640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d806fcd0 0x7f87d814ee30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00b410 tx=0x7f87cc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87bffff640 1 -- 172.21.15.77:0/3503397422 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc015020 con 0x7f87d806fcd0 2024-04-25T01:23:37.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87bffff640 1 -- 172.21.15.77:0/3503397422 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87cc011850 con 0x7f87d806fcd0 2024-04-25T01:23:37.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.892+0000 7f87dddc8640 1 -- 172.21.15.77:0/3503397422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d814d380 con 0x7f87d806fcd0 2024-04-25T01:23:37.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.893+0000 7f87bffff640 1 -- 172.21.15.77:0/3503397422 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc011c40 con 0x7f87d806fcd0 2024-04-25T01:23:37.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.893+0000 7f87dddc8640 1 -- 172.21.15.77:0/3503397422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d814d940 con 0x7f87d806fcd0 2024-04-25T01:23:37.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.894+0000 7f87dddc8640 1 -- 172.21.15.77:0/3503397422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87d8068fb0 con 0x7f87d806fcd0 2024-04-25T01:23:37.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.894+0000 7f87bffff640 1 -- 172.21.15.77:0/3503397422 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f87cc018020 con 0x7f87d806fcd0 2024-04-25T01:23:37.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.894+0000 7f87bffff640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 0x7f87c406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.894+0000 7f87bffff640 1 -- 172.21.15.77:0/3503397422 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f87cc08ae90 con 0x7f87d806fcd0 2024-04-25T01:23:37.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.894+0000 7f87d77fe640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 0x7f87c406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.895+0000 7f87d77fe640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 0x7f87c406e2f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f87c8034000 tx=0x7f87c802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:37.899+0000 7f87bffff640 1 -- 172.21.15.77:0/3503397422 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f87cc01f020 con 0x7f87d806fcd0 2024-04-25T01:23:38.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.108+0000 7f87dddc8640 1 -- 172.21.15.77:0/3503397422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f87d80675a0 con 0x7f87d806fcd0 2024-04-25T01:23:38.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.108+0000 7f87bffff640 1 -- 172.21.15.77:0/3503397422 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7f87cc0172c0 con 0x7f87d806fcd0 2024-04-25T01:23:38.089 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T01:23:38.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 -- 172.21.15.77:0/3503397422 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 msgr2=0x7f87c406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:38.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 0x7f87c406e2f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f87c8034000 tx=0x7f87c802e000 comp rx=0 tx=0).stop 2024-04-25T01:23:38.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 -- 172.21.15.77:0/3503397422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d806fcd0 msgr2=0x7f87d814ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:38.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d806fcd0 0x7f87d814ee30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00b410 tx=0x7f87cc00b8d0 comp rx=0 tx=0).stop 2024-04-25T01:23:38.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 -- 172.21.15.77:0/3503397422 shutdown_connections 2024-04-25T01:23:38.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 0x7f87c406e2f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:38.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d806fcd0 0x7f87d814ee30 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:38.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d80ade10 0x7f87d814f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:38.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 --2- 172.21.15.77:0/3503397422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d806c670 0x7f87d80ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:38.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 -- 172.21.15.77:0/3503397422 >> 172.21.15.77:0/3503397422 conn(0x7f87d8071550 msgr2=0x7f87d8153fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:38.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 -- 172.21.15.77:0/3503397422 shutdown_connections 2024-04-25T01:23:38.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38.110+0000 7f87dddc8640 1 -- 172.21.15.77:0/3503397422 wait complete. 2024-04-25T01:23:38.102 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T01:23:38.102 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:23:38.102 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-04-25T01:23:38.102 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:23:38.102 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:38.102 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T01:23:38.102 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T01:23:38.102 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:23:38.102 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:23:38.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:38.103 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-04-25T01:23:38.118 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:23:38.118 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:23:38.118 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:23:38.118 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:38.118 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-04-25T01:23:38.118 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:38.118 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42630sec preferred_lft 42630sec 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a9ad:d2f:d456:e65f/64 scope link noprefixroute 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0a:90:ec:3f:d1:57 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::b40c:78ff:fecf:cd0c/64 scope link 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0a:90:ec:3f:d1:57 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::890:ecff:fe3f:d157/64 scope link 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 12:78:6c:75:3b:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1078:6cff:fe75:3be8/64 scope link 2024-04-25T01:23:38.120 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:38.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:38.121 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:23:38.181 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-04-25T01:23:38.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:38.183 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-04-25T01:23:38.237 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-04-25T01:23:38.238 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:23:38.238 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:23:38.238 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:23:38.238 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:23:38.238 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:23:38.238 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-04-25T01:23:38.239 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:23:38.239 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:23:38.239 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:23:38.239 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-04-25T01:23:38.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:23:38.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:38.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:38.361 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-04-25T01:23:38.375 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T01:23:38.375 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:23:38.376 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:23:38.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:38.376 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-04-25T01:23:38.439 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:23:38.439 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:38.494 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:23:38.494 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:23:38.494 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:38.549 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:23:38.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:38.549 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:38.629 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:38.629 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:23:38.629 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:23:38.637 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:38.638 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:23:38.638 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:23:38.638 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:23:38.638 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:23:38.643 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:23:38.643 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:23:38.643 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-04-25T01:23:38.643 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.121:3300,172.21.15.121:3301" 2024-04-25T01:23:38.643 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:23:38.643 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790" 2024-04-25T01:23:38.643 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.121:6789,172.21.15.121:6790:/ 2024-04-25T01:23:38.643 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=7c3c39ee-077e-476c-8994-eede4fab301d" will pass to kernel 2024-04-25T01:23:38.646 INFO:teuthology.orchestra.run.smithi077.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T01:23:38.646 INFO:teuthology.orchestra.run.smithi077.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:23:38.646 INFO:teuthology.orchestra.run.smithi077.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:23:38.646 INFO:teuthology.orchestra.run.smithi077.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:23:38.646 INFO:teuthology.orchestra.run.smithi077.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:23:38.647 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:23:38.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:38.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:23:38.711 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:23:38.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:23:38.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:38.808 INFO:teuthology.orchestra.run.smithi077.stdout:client5087 2024-04-25T01:23:38.808 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:23:38.808 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:38.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:38.823 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch test_file1) 2024-04-25T01:23:38.881 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:23:38.881 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:38.937 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-04-25T01:23:38.937 DEBUG:teuthology.orchestra.run.smithi077:> import fcntl 2024-04-25T01:23:38.937 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> import sys 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> import time 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> fd = os.open("/home/ubuntu/cephtest/mnt.0/background_file", os.O_RDWR | os.O_CREAT, 0o644) 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> try: 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> while True: 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> print("write_background: writing", file=sys.stderr) 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> os.write(fd, b'"'"'content'"'"') 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> if not True: 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> break 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> try: 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> if os.read(0, 4096) == b"": 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> break 2024-04-25T01:23:38.938 DEBUG:teuthology.orchestra.run.smithi077:> except BlockingIOError: 2024-04-25T01:23:38.939 DEBUG:teuthology.orchestra.run.smithi077:> pass 2024-04-25T01:23:38.939 DEBUG:teuthology.orchestra.run.smithi077:> time.sleep(2) 2024-04-25T01:23:38.939 DEBUG:teuthology.orchestra.run.smithi077:> except IOError as e: 2024-04-25T01:23:38.939 DEBUG:teuthology.orchestra.run.smithi077:> pass 2024-04-25T01:23:38.939 DEBUG:teuthology.orchestra.run.smithi077:> os.close(fd) 2024-04-25T01:23:38.939 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-04-25T01:23:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:write_background: writing 2024-04-25T01:23:41.045 INFO:teuthology.orchestra.run.smithi077.stderr:write_background: writing 2024-04-25T01:23:43.047 INFO:teuthology.orchestra.run.smithi077.stderr:write_background: writing 2024-04-25T01:23:45.049 INFO:teuthology.orchestra.run.smithi077.stderr:write_background: writing 2024-04-25T01:23:45.836 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:23:45.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:23:47.050 INFO:teuthology.orchestra.run.smithi077.stderr:write_background: writing 2024-04-25T01:23:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:write_background: writing 2024-04-25T01:23:49.081 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:49.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.208+0000 7f2674f63640 1 Processor -- start 2024-04-25T01:23:49.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.208+0000 7f2674f63640 1 -- start start 2024-04-25T01:23:49.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.209+0000 7f2674f63640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f267015e4c0 0x7f26700d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:49.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.209+0000 7f2674f63640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f26700d4a10 0x7f26700b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:49.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.209+0000 7f2674f63640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26700d1e90 con 0x7f26700d2d40 2024-04-25T01:23:49.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.209+0000 7f2674f63640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f26700b54a0 con 0x7f267015e4c0 2024-04-25T01:23:49.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.209+0000 7f2674f63640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f26700d2130 con 0x7f26700d4a10 2024-04-25T01:23:49.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.209+0000 7f266e59c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f267015e4c0 0x7f26700d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:49.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.209+0000 7f266dd9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f26700d4a10 0x7f26700b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:49.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.209+0000 7f266e59c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f267015e4c0 0x7f26700d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:32768/0 (socket says 172.21.15.77:32768) 2024-04-25T01:23:49.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.209+0000 7f266e59c640 1 -- 172.21.15.77:0/3348013297 learned_addr learned my addr 172.21.15.77:0/3348013297 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:49.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.209+0000 7f266ed9d640 1 --1- 172.21.15.77:0/3348013297 >> v1:172.21.15.121:6789/0 conn(0x7f26700d2d40 0x7f26700d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:54828/0 (socket says 172.21.15.77:54828) 2024-04-25T01:23:49.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.210+0000 7f266e59c640 1 -- 172.21.15.77:0/3348013297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f26700d4a10 msgr2=0x7f26700b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:49.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.210+0000 7f266e59c640 1 --2- 172.21.15.77:0/3348013297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f26700d4a10 0x7f26700b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:49.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.210+0000 7f266e59c640 1 -- 172.21.15.77:0/3348013297 >> v1:172.21.15.121:6789/0 conn(0x7f26700d2d40 legacy=0x7f26700d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:49.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.210+0000 7f266e59c640 1 -- 172.21.15.77:0/3348013297 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26700b5780 con 0x7f267015e4c0 2024-04-25T01:23:49.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.210+0000 7f266dd9b640 1 --2- 172.21.15.77:0/3348013297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f26700d4a10 0x7f26700b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:49.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f266e59c640 1 --2- 172.21.15.77:0/3348013297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f267015e4c0 0x7f26700d2800 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2660009870 tx=0x7f266002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74b7eb2bda833eb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:49.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f266d59a640 1 -- 172.21.15.77:0/3348013297 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660037070 con 0x7f267015e4c0 2024-04-25T01:23:49.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f266d59a640 1 -- 172.21.15.77:0/3348013297 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2660032440 con 0x7f267015e4c0 2024-04-25T01:23:49.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f266d59a640 1 -- 172.21.15.77:0/3348013297 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660032810 con 0x7f267015e4c0 2024-04-25T01:23:49.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f2674f63640 1 -- 172.21.15.77:0/3348013297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f267015e4c0 msgr2=0x7f26700d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:49.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f2674f63640 1 --2- 172.21.15.77:0/3348013297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f267015e4c0 0x7f26700d2800 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2660009870 tx=0x7f266002a1c0 comp rx=0 tx=0).stop 2024-04-25T01:23:49.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f2674f63640 1 -- 172.21.15.77:0/3348013297 shutdown_connections 2024-04-25T01:23:49.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f2674f63640 1 --2- 172.21.15.77:0/3348013297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f26700d4a10 0x7f26700b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:49.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f2674f63640 1 --2- 172.21.15.77:0/3348013297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f267015e4c0 0x7f26700d2800 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:49.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f2674f63640 1 -- 172.21.15.77:0/3348013297 >> 172.21.15.77:0/3348013297 conn(0x7f26700d3ef0 msgr2=0x7f26700d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:49.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f2674f63640 1 -- 172.21.15.77:0/3348013297 shutdown_connections 2024-04-25T01:23:49.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.211+0000 7f2674f63640 1 -- 172.21.15.77:0/3348013297 wait complete. 2024-04-25T01:23:49.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.213+0000 7f2674f63640 1 Processor -- start 2024-04-25T01:23:49.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.213+0000 7f2674f63640 1 -- start start 2024-04-25T01:23:49.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.213+0000 7f2674f63640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26700d2d40 0x7f2670151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:49.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.213+0000 7f2674f63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26700d4a10 0x7f26700b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:49.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.213+0000 7f2674f63640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f267015e4c0 0x7f26700b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:49.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.213+0000 7f2674f63640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f267015ea20 con 0x7f26700d4a10 2024-04-25T01:23:49.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.213+0000 7f2674f63640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f267015e8a0 con 0x7f26700d2d40 2024-04-25T01:23:49.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.213+0000 7f2674f63640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f26700d1010 con 0x7f267015e4c0 2024-04-25T01:23:49.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26700d4a10 0x7f26700b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26700d4a10 0x7f26700b0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53256/0 (socket says 172.21.15.77:53256) 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266dd9b640 1 -- 172.21.15.77:0/714097937 learned_addr learned my addr 172.21.15.77:0/714097937 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266e59c640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26700d2d40 0x7f2670151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266ed9d640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f267015e4c0 0x7f26700b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266dd9b640 1 -- 172.21.15.77:0/714097937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f267015e4c0 msgr2=0x7f26700b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266dd9b640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f267015e4c0 0x7f26700b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266dd9b640 1 -- 172.21.15.77:0/714097937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26700d2d40 msgr2=0x7f2670151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266dd9b640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26700d2d40 0x7f2670151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266dd9b640 1 -- 172.21.15.77:0/714097937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26700b1520 con 0x7f26700d4a10 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266e59c640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26700d2d40 0x7f2670151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f266dd9b640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26700d4a10 0x7f26700b0e20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f266400e930 tx=0x7f266400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f265effd640 1 -- 172.21.15.77:0/714097937 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2664017070 con 0x7f26700d4a10 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f265effd640 1 -- 172.21.15.77:0/714097937 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f266400ca00 con 0x7f26700d4a10 2024-04-25T01:23:49.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f2674f63640 1 -- 172.21.15.77:0/714097937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26700b1800 con 0x7f26700d4a10 2024-04-25T01:23:49.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f265effd640 1 -- 172.21.15.77:0/714097937 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266400cdf0 con 0x7f26700d4a10 2024-04-25T01:23:49.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.214+0000 7f2674f63640 1 -- 172.21.15.77:0/714097937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26701f4ae0 con 0x7f26700d4a10 2024-04-25T01:23:49.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.215+0000 7f2674f63640 1 -- 172.21.15.77:0/714097937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2670068fb0 con 0x7f26700d4a10 2024-04-25T01:23:49.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.216+0000 7f265effd640 1 -- 172.21.15.77:0/714097937 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f266401a070 con 0x7f26700d4a10 2024-04-25T01:23:49.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.216+0000 7f265effd640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f265006be30 0x7f265006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:49.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.216+0000 7f265effd640 1 -- 172.21.15.77:0/714097937 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f266408ab00 con 0x7f26700d4a10 2024-04-25T01:23:49.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.216+0000 7f266e59c640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f265006be30 0x7f265006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:49.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.217+0000 7f266e59c640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f265006be30 0x7f265006e2f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2660034000 tx=0x7f266002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:49.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.220+0000 7f265effd640 1 -- 172.21.15.77:0/714097937 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2664058c50 con 0x7f26700d4a10 2024-04-25T01:23:49.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.427+0000 7f2674f63640 1 -- 172.21.15.77:0/714097937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26700cad10 con 0x7f26700d4a10 2024-04-25T01:23:49.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.428+0000 7f265effd640 1 -- 172.21.15.77:0/714097937 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5184 (secure 0 0 0) 0x7f266405cf50 con 0x7f26700d4a10 2024-04-25T01:23:49.409 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:49.409 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5254,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/865504322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":865504322},{"type":"v1","addr":"172.21.15.121:6835","nonce":865504322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":5886,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3189776231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3189776231},{"type":"v1","addr":"172.21.15.77:6836","nonce":3189776231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3454287131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3454287131},{"type":"v1","addr":"172.21.15.77:6837","nonce":3454287131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:23:28.170422+0000","modified":"2024-04-25T01:23:30.191645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6834/3834135103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3834135103},{"type":"v1","addr":"172.21.15.121:6834","nonce":3834135103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":3}]} 2024-04-25T01:23:49.410 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 27 2024-04-25T01:23:49.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 -- 172.21.15.77:0/714097937 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f265006be30 msgr2=0x7f265006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:49.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f265006be30 0x7f265006e2f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2660034000 tx=0x7f266002e000 comp rx=0 tx=0).stop 2024-04-25T01:23:49.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 -- 172.21.15.77:0/714097937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26700d4a10 msgr2=0x7f26700b0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:49.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26700d4a10 0x7f26700b0e20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f266400e930 tx=0x7f266400edf0 comp rx=0 tx=0).stop 2024-04-25T01:23:49.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 -- 172.21.15.77:0/714097937 shutdown_connections 2024-04-25T01:23:49.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f265006be30 0x7f265006e2f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:49.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26700d4a10 0x7f26700b0e20 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:49.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f267015e4c0 0x7f26700b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:49.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 --2- 172.21.15.77:0/714097937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26700d2d40 0x7f2670151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:49.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 -- 172.21.15.77:0/714097937 >> 172.21.15.77:0/714097937 conn(0x7f26700d3ef0 msgr2=0x7f26700b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:49.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 -- 172.21.15.77:0/714097937 shutdown_connections 2024-04-25T01:23:49.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:49.430+0000 7f2674f63640 1 -- 172.21.15.77:0/714097937 wait complete. 2024-04-25T01:23:49.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json force_readonly 2024-04-25T01:23:49.547 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-04-25T01:23:51.054 INFO:teuthology.orchestra.run.smithi077.stderr:write_background: writing 2024-04-25T01:23:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:59.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:59.548 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch test_file1) 2024-04-25T01:23:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:23:59.568 INFO:teuthology.orchestra.run.smithi077.stderr:touch: setting times of 'test_file1': Read-only file system 2024-04-25T01:23:59.568 INFO:tasks.ceph_test_case:waiting 30s for health warning matching MDS_READ_ONLY 2024-04-25T01:23:59.568 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T01:23:59.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.733+0000 7fe7e4de9640 1 Processor -- start 2024-04-25T01:23:59.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.733+0000 7fe7e4de9640 1 -- start start 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.734+0000 7fe7e4de9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe7e0159cc0 0x7fe7e0072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.734+0000 7fe7e4de9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7e006c0f0 0x7fe7e0072e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.734+0000 7fe7e4de9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7e0071ce0 con 0x7fe7e0066ea0 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.734+0000 7fe7e4de9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fe7e015d530 con 0x7fe7e006c0f0 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.734+0000 7fe7e4de9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe7e015d000 con 0x7fe7e0159cc0 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.734+0000 7fe7ded9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7e006c0f0 0x7fe7e0072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.734+0000 7fe7ded9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7e006c0f0 0x7fe7e0072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45346/0 (socket says 172.21.15.77:45346) 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.734+0000 7fe7ded9d640 1 -- 172.21.15.77:0/4289591646 learned_addr learned my addr 172.21.15.77:0/4289591646 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.735+0000 7fe7ded9d640 1 -- 172.21.15.77:0/4289591646 >> v2:172.21.15.121:3300/0 conn(0x7fe7e0159cc0 msgr2=0x7fe7e0072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.735+0000 7fe7de59c640 1 --2- 172.21.15.77:0/4289591646 >> v2:172.21.15.121:3300/0 conn(0x7fe7e0159cc0 0x7fe7e0072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.735+0000 7fe7ded9d640 1 --2- 172.21.15.77:0/4289591646 >> v2:172.21.15.121:3300/0 conn(0x7fe7e0159cc0 0x7fe7e0072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.735+0000 7fe7ded9d640 1 -- 172.21.15.77:0/4289591646 >> v1:172.21.15.121:6789/0 conn(0x7fe7e0066ea0 legacy=0x7fe7e015fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.735+0000 7fe7ded9d640 1 -- 172.21.15.77:0/4289591646 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e015d250 con 0x7fe7e006c0f0 2024-04-25T01:23:59.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.735+0000 7fe7de59c640 1 --2- 172.21.15.77:0/4289591646 >> v2:172.21.15.121:3300/0 conn(0x7fe7e0159cc0 0x7fe7e0072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.735+0000 7fe7ded9d640 1 --2- 172.21.15.77:0/4289591646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e006c0f0 0x7fe7e0072e40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00b5f40 tx=0x7fe7d002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9352a9c3e6bb6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.735+0000 7fe7dd59a640 1 -- 172.21.15.77:0/4289591646 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0037070 con 0x7fe7e006c0f0 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.736+0000 7fe7dd59a640 1 -- 172.21.15.77:0/4289591646 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d0032440 con 0x7fe7e006c0f0 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.736+0000 7fe7dd59a640 1 -- 172.21.15.77:0/4289591646 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0032810 con 0x7fe7e006c0f0 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.736+0000 7fe7e4de9640 1 -- 172.21.15.77:0/4289591646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e006c0f0 msgr2=0x7fe7e0072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.736+0000 7fe7e4de9640 1 --2- 172.21.15.77:0/4289591646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e006c0f0 0x7fe7e0072e40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00b5f40 tx=0x7fe7d002a230 comp rx=0 tx=0).stop 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.736+0000 7fe7e4de9640 1 -- 172.21.15.77:0/4289591646 shutdown_connections 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.736+0000 7fe7e4de9640 1 --2- 172.21.15.77:0/4289591646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e006c0f0 0x7fe7e0072e40 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.736+0000 7fe7e4de9640 1 --2- 172.21.15.77:0/4289591646 >> v2:172.21.15.121:3300/0 conn(0x7fe7e0159cc0 0x7fe7e0072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.736+0000 7fe7e4de9640 1 -- 172.21.15.77:0/4289591646 >> 172.21.15.77:0/4289591646 conn(0x7fe7e00b5a40 msgr2=0x7fe7e00b5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.736+0000 7fe7e4de9640 1 -- 172.21.15.77:0/4289591646 shutdown_connections 2024-04-25T01:23:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.736+0000 7fe7e4de9640 1 -- 172.21.15.77:0/4289591646 wait complete. 2024-04-25T01:23:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7e4de9640 1 Processor -- start 2024-04-25T01:23:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7e4de9640 1 -- start start 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7e4de9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7e0066ea0 0x7fe7e0148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7e4de9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7e006c0f0 0x7fe7e01f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7e4de9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e0159cc0 0x7fe7e0072b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7e4de9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e006bbd0 con 0x7fe7e0159cc0 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7e4de9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e00afde0 con 0x7fe7e0066ea0 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7e4de9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe7e00b3e80 con 0x7fe7e006c0f0 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7ded9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e0159cc0 0x7fe7e0072b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7ded9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e0159cc0 0x7fe7e0072b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45352/0 (socket says 172.21.15.77:45352) 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7ded9d640 1 -- 172.21.15.77:0/701322853 learned_addr learned my addr 172.21.15.77:0/701322853 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.738+0000 7fe7de59c640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7e0066ea0 0x7fe7e0148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7ddd9b640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7e006c0f0 0x7fe7e01f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7ded9d640 1 -- 172.21.15.77:0/701322853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7e006c0f0 msgr2=0x7fe7e01f5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7ded9d640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7e006c0f0 0x7fe7e01f5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7ded9d640 1 -- 172.21.15.77:0/701322853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7e0066ea0 msgr2=0x7fe7e0148780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7ded9d640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7e0066ea0 0x7fe7e0148780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7ded9d640 1 -- 172.21.15.77:0/701322853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e01f6020 con 0x7fe7e0159cc0 2024-04-25T01:23:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7ded9d640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e0159cc0 0x7fe7e0072b70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00090b0 tx=0x7fe7d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7d6ffd640 1 -- 172.21.15.77:0/701322853 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0041020 con 0x7fe7e0159cc0 2024-04-25T01:23:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7d6ffd640 1 -- 172.21.15.77:0/701322853 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d0030990 con 0x7fe7e0159cc0 2024-04-25T01:23:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7e4de9640 1 -- 172.21.15.77:0/701322853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e01544b0 con 0x7fe7e0159cc0 2024-04-25T01:23:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7d6ffd640 1 -- 172.21.15.77:0/701322853 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0037040 con 0x7fe7e0159cc0 2024-04-25T01:23:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.739+0000 7fe7e4de9640 1 -- 172.21.15.77:0/701322853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e0154a70 con 0x7fe7e0159cc0 2024-04-25T01:23:59.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.740+0000 7fe7e4de9640 1 -- 172.21.15.77:0/701322853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7e0068fb0 con 0x7fe7e0159cc0 2024-04-25T01:23:59.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.740+0000 7fe7d6ffd640 1 -- 172.21.15.77:0/701322853 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe7d0044020 con 0x7fe7e0159cc0 2024-04-25T01:23:59.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.741+0000 7fe7d6ffd640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7c006be60 0x7fe7c006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.741+0000 7fe7d6ffd640 1 -- 172.21.15.77:0/701322853 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fe7d00ab4e0 con 0x7fe7e0159cc0 2024-04-25T01:23:59.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.741+0000 7fe7de59c640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7c006be60 0x7fe7c006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.741+0000 7fe7de59c640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7c006be60 0x7fe7c006e320 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe7d800f500 tx=0x7fe7d8007770 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:59.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.745+0000 7fe7d6ffd640 1 -- 172.21.15.77:0/701322853 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe7d004b070 con 0x7fe7e0159cc0 2024-04-25T01:23:59.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.958+0000 7fe7e4de9640 1 -- 172.21.15.77:0/701322853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe7e00735e0 con 0x7fe7e0159cc0 2024-04-25T01:23:59.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.958+0000 7fe7d6ffd640 1 -- 172.21.15.77:0/701322853 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+159 (secure 0 0 0) 0x7fe7e01f6020 con 0x7fe7e0159cc0 2024-04-25T01:23:59.938 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:23:59.938 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_WARN","checks":{"MDS_READ_ONLY":{"severity":"HEALTH_WARN","summary":{"message":"1 MDSs are read only","count":1},"muted":false}},"mutes":[]} 2024-04-25T01:23:59.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 -- 172.21.15.77:0/701322853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7c006be60 msgr2=0x7fe7c006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7c006be60 0x7fe7c006e320 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe7d800f500 tx=0x7fe7d8007770 comp rx=0 tx=0).stop 2024-04-25T01:23:59.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 -- 172.21.15.77:0/701322853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e0159cc0 msgr2=0x7fe7e0072b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e0159cc0 0x7fe7e0072b70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00090b0 tx=0x7fe7d00046c0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 -- 172.21.15.77:0/701322853 shutdown_connections 2024-04-25T01:23:59.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe7c006be60 0x7fe7c006e320 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e0159cc0 0x7fe7e0072b70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7e006c0f0 0x7fe7e01f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 --2- 172.21.15.77:0/701322853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7e0066ea0 0x7fe7e0148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 -- 172.21.15.77:0/701322853 >> 172.21.15.77:0/701322853 conn(0x7fe7e00b5a40 msgr2=0x7fe7e00cf5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 -- 172.21.15.77:0/701322853 shutdown_connections 2024-04-25T01:23:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:23:59.960+0000 7fe7e4de9640 1 -- 172.21.15.77:0/701322853 wait complete. 2024-04-25T01:23:59.950 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-25T01:23:59.951 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T01:23:59.951 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:23:59.951 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T01:23:59.951 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:23:59.952 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T01:23:59.952 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:23:59.952 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T01:23:59.952 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:23:59.961 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T01:23:59.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-04-25T01:23:59.962 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T01:23:59.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-04-25T01:23:59.965 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T01:23:59.965 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-04-25T01:23:59.966 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T01:23:59.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-04-25T01:24:00.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.124+0000 7f5e5b87d640 1 Processor -- start 2024-04-25T01:24:00.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.124+0000 7fe58fb5a640 1 Processor -- start 2024-04-25T01:24:00.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.125+0000 7fde9a790640 1 Processor -- start 2024-04-25T01:24:00.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.125+0000 7feb6db6a640 1 Processor -- start 2024-04-25T01:24:00.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.125+0000 7f5e5b87d640 1 -- start start 2024-04-25T01:24:00.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.125+0000 7feb6db6a640 1 -- start start 2024-04-25T01:24:00.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.125+0000 7fde9a790640 1 -- start start 2024-04-25T01:24:00.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.125+0000 7fe58fb5a640 1 -- start start 2024-04-25T01:24:00.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.125+0000 7feb6db6a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feb6815f630 0x7feb680ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fe58fb5a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe588143940 0x7fe58806e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb6db6a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7feb680d28a0 0x7feb680b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7f5e5b87d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5e54143940 0x7f5e5406e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fe58fb5a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe58815f400 0x7fe58806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fe58fb5a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe588070e60 con 0x7fe588143d20 2024-04-25T01:24:00.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fde9a790640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde9415e4c0 0x7fde940d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fde9a790640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde940d2d40 0x7fde940cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fde9a790640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde940d1e90 con 0x7fde940d4a10 2024-04-25T01:24:00.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fde9a790640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fde940b54a0 con 0x7fde940d2d40 2024-04-25T01:24:00.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fde9a790640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fde940d2130 con 0x7fde9415e4c0 2024-04-25T01:24:00.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fe58fb5a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe588071140 con 0x7fe58815f400 2024-04-25T01:24:00.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fe58fb5a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe588071420 con 0x7fe588143940 2024-04-25T01:24:00.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb6db6a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb680af340 con 0x7feb680b0920 2024-04-25T01:24:00.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb6db6a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7feb680af620 con 0x7feb6815f630 2024-04-25T01:24:00.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb6db6a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7feb680af900 con 0x7feb680d28a0 2024-04-25T01:24:00.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb677fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feb6815f630 0x7feb680ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb677fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feb6815f630 0x7feb680ae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45366/0 (socket says 172.21.15.77:45366) 2024-04-25T01:24:00.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7f5e5b87d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e54070e60 con 0x7f5e54143d20 2024-04-25T01:24:00.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb677fe640 1 -- 172.21.15.77:0/358850826 learned_addr learned my addr 172.21.15.77:0/358850826 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:00.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7f5e5b87d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e54071140 con 0x7f5e5415f400 2024-04-25T01:24:00.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fe58d0f5640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe588143d20 0x7fe58815ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41118/0 (socket says 172.21.15.77:41118) 2024-04-25T01:24:00.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fe58d0f5640 1 -- 172.21.15.77:0/3950125420 learned_addr learned my addr 172.21.15.77:0/3950125420 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:00.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7f5e5b87d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f5e54071420 con 0x7f5e54143940 2024-04-25T01:24:00.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fe58d8f6640 1 --2- 172.21.15.77:0/3950125420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe588143940 0x7fe58806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7f5e58e18640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5e54143d20 0x7f5e5415ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41132/0 (socket says 172.21.15.77:41132) 2024-04-25T01:24:00.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7f5e58e18640 1 -- 172.21.15.77:0/2987649598 learned_addr learned my addr 172.21.15.77:0/2987649598 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:00.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7f5e59619640 1 --2- 172.21.15.77:0/2987649598 >> v2:172.21.15.77:3300/0 conn(0x7f5e54143940 0x7f5e5406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fde98d2d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde940d2d40 0x7fde940cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fde937fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fde940d4a10 0x7fde940b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:39660/0 (socket says 172.21.15.77:39660) 2024-04-25T01:24:00.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fde937fe640 1 -- 172.21.15.77:0/135022163 learned_addr learned my addr 172.21.15.77:0/135022163 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:00.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fde93fff640 1 --2- 172.21.15.77:0/135022163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde9415e4c0 0x7fde940d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fe58e0f7640 1 --2- 172.21.15.77:0/3950125420 >> v2:172.21.15.121:3300/0 conn(0x7fe58815f400 0x7fe58806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fe58c8f4640 1 -- 172.21.15.77:0/3950125420 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704371325 0 0) 0x7fe588070e60 con 0x7fe588143d20 2024-04-25T01:24:00.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fe58c8f4640 1 -- 172.21.15.77:0/3950125420 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe574003520 con 0x7fe588143d20 2024-04-25T01:24:00.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7f5e4bfff640 1 -- 172.21.15.77:0/2987649598 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 755233760 0 0) 0x7f5e54070e60 con 0x7f5e54143d20 2024-04-25T01:24:00.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb66ffd640 1 --2- 172.21.15.77:0/358850826 >> v2:172.21.15.121:3300/0 conn(0x7feb680d28a0 0x7feb680b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb677fe640 1 -- 172.21.15.77:0/358850826 >> v2:172.21.15.121:3300/0 conn(0x7feb680d28a0 msgr2=0x7feb680b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:00.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb677fe640 1 --2- 172.21.15.77:0/358850826 >> v2:172.21.15.121:3300/0 conn(0x7feb680d28a0 0x7feb680b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb677fe640 1 -- 172.21.15.77:0/358850826 >> v1:172.21.15.121:6789/0 conn(0x7feb680b0920 legacy=0x7feb680b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7feb677fe640 1 -- 172.21.15.77:0/358850826 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb680afbe0 con 0x7feb6815f630 2024-04-25T01:24:00.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7fe58d8f6640 1 -- 172.21.15.77:0/3950125420 >> v2:172.21.15.121:3300/0 conn(0x7fe58815f400 msgr2=0x7fe58806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:00.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7f5e4bfff640 1 -- 172.21.15.77:0/2987649598 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e40003520 con 0x7f5e54143d20 2024-04-25T01:24:00.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7fe58d8f6640 1 --2- 172.21.15.77:0/3950125420 >> v2:172.21.15.121:3300/0 conn(0x7fe58815f400 0x7fe58806eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7fe58d8f6640 1 -- 172.21.15.77:0/3950125420 >> v1:172.21.15.77:6789/0 conn(0x7fe588143d20 legacy=0x7fe58815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7fe58c8f4640 1 -- 172.21.15.77:0/3950125420 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1150236031 0 0) 0x7fe574003520 con 0x7fe588143d20 2024-04-25T01:24:00.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7fe58d8f6640 1 -- 172.21.15.77:0/3950125420 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe588071700 con 0x7fe588143940 2024-04-25T01:24:00.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7f5e59619640 1 -- 172.21.15.77:0/2987649598 >> v1:172.21.15.121:6789/0 conn(0x7f5e5415f400 legacy=0x7f5e54071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7f5e59619640 1 -- 172.21.15.77:0/2987649598 >> v1:172.21.15.77:6789/0 conn(0x7f5e54143d20 legacy=0x7f5e5415ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7f5e59619640 1 -- 172.21.15.77:0/2987649598 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e54071700 con 0x7f5e54143940 2024-04-25T01:24:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.126+0000 7fde92ffd640 1 -- 172.21.15.77:0/135022163 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3624935305 0 0) 0x7fde940d1e90 con 0x7fde940d4a10 2024-04-25T01:24:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7fde92ffd640 1 -- 172.21.15.77:0/135022163 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde7c003520 con 0x7fde940d4a10 2024-04-25T01:24:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7fde98d2d640 1 -- 172.21.15.77:0/135022163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde9415e4c0 msgr2=0x7fde940d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7fde98d2d640 1 --2- 172.21.15.77:0/135022163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde9415e4c0 0x7fde940d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7fde98d2d640 1 -- 172.21.15.77:0/135022163 >> v1:172.21.15.121:6789/0 conn(0x7fde940d4a10 legacy=0x7fde940b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7fde98d2d640 1 -- 172.21.15.77:0/135022163 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde940b5780 con 0x7fde940d2d40 2024-04-25T01:24:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7f5e59619640 1 --2- 172.21.15.77:0/2987649598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e54143940 0x7f5e5406e650 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5e500097d0 tx=0x7f5e5002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4fc7b13840f212 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7fe58d8f6640 1 --2- 172.21.15.77:0/3950125420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe588143940 0x7fe58806e650 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe5840097a0 tx=0x7fe58402a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15dee70f62ac112 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.127+0000 7feb677fe640 1 --2- 172.21.15.77:0/358850826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb6815f630 0x7feb680ae9f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7feb680720d0 tx=0x7feb5802e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3367c72cf6ed58b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7fe58c8f4640 1 -- 172.21.15.77:0/3950125420 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe584037070 con 0x7fe588143940 2024-04-25T01:24:00.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7f5e4bfff640 1 -- 172.21.15.77:0/2987649598 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50037070 con 0x7f5e54143940 2024-04-25T01:24:00.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7feb667fc640 1 -- 172.21.15.77:0/358850826 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5803a070 con 0x7feb6815f630 2024-04-25T01:24:00.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7fde98d2d640 1 --2- 172.21.15.77:0/135022163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde940d2d40 0x7fde940cfaf0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fde8c009d90 tx=0x7fde8c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7add9369667d5f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7f5e4bfff640 1 -- 172.21.15.77:0/2987649598 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e50032440 con 0x7f5e54143940 2024-04-25T01:24:00.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7f5e4bfff640 1 -- 172.21.15.77:0/2987649598 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e500327f0 con 0x7f5e54143940 2024-04-25T01:24:00.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7fe58c8f4640 1 -- 172.21.15.77:0/3950125420 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe58402aba0 con 0x7fe588143940 2024-04-25T01:24:00.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7fe58c8f4640 1 -- 172.21.15.77:0/3950125420 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe578001670 con 0x7fe588143940 2024-04-25T01:24:00.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7feb667fc640 1 -- 172.21.15.77:0/358850826 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb58035600 con 0x7feb6815f630 2024-04-25T01:24:00.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7feb667fc640 1 -- 172.21.15.77:0/358850826 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb580359d0 con 0x7feb6815f630 2024-04-25T01:24:00.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7fde92ffd640 1 -- 172.21.15.77:0/135022163 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c031600 con 0x7fde940d2d40 2024-04-25T01:24:00.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7fde92ffd640 1 -- 172.21.15.77:0/135022163 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde8c0317a0 con 0x7fde940d2d40 2024-04-25T01:24:00.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.128+0000 7fde92ffd640 1 -- 172.21.15.77:0/135022163 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c031b70 con 0x7fde940d2d40 2024-04-25T01:24:00.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7f5e5b87d640 1 -- 172.21.15.77:0/2987649598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e54143940 msgr2=0x7f5e5406e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7f5e5b87d640 1 --2- 172.21.15.77:0/2987649598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e54143940 0x7f5e5406e650 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5e500097d0 tx=0x7f5e5002a270 comp rx=0 tx=0).stop 2024-04-25T01:24:00.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7f5e5b87d640 1 -- 172.21.15.77:0/2987649598 shutdown_connections 2024-04-25T01:24:00.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3950125420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe588143940 msgr2=0x7fe58806e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fde9a790640 1 -- 172.21.15.77:0/135022163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde940d2d40 msgr2=0x7fde940cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fde9a790640 1 --2- 172.21.15.77:0/135022163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde940d2d40 0x7fde940cfaf0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fde8c009d90 tx=0x7fde8c02a5d0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7f5e5b87d640 1 --2- 172.21.15.77:0/2987649598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e54143940 0x7f5e5406e650 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fe58fb5a640 1 --2- 172.21.15.77:0/3950125420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe588143940 0x7fe58806e650 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe5840097a0 tx=0x7fe58402a110 comp rx=0 tx=0).stop 2024-04-25T01:24:00.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7f5e5b87d640 1 -- 172.21.15.77:0/2987649598 >> 172.21.15.77:0/2987649598 conn(0x7f5e540b1820 msgr2=0x7f5e540b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:00.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7feb6db6a640 1 -- 172.21.15.77:0/358850826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb6815f630 msgr2=0x7feb680ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3950125420 shutdown_connections 2024-04-25T01:24:00.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fde9a790640 1 -- 172.21.15.77:0/135022163 shutdown_connections 2024-04-25T01:24:00.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fe58fb5a640 1 --2- 172.21.15.77:0/3950125420 >> v2:172.21.15.121:3300/0 conn(0x7fe58815f400 0x7fe58806eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fe58fb5a640 1 --2- 172.21.15.77:0/3950125420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe588143940 0x7fe58806e650 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fde9a790640 1 --2- 172.21.15.77:0/135022163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde940d2d40 0x7fde940cfaf0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fde9a790640 1 --2- 172.21.15.77:0/135022163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde9415e4c0 0x7fde940d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fde9a790640 1 -- 172.21.15.77:0/135022163 >> 172.21.15.77:0/135022163 conn(0x7fde940d3ef0 msgr2=0x7fde940d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:00.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7f5e5b87d640 1 -- 172.21.15.77:0/2987649598 shutdown_connections 2024-04-25T01:24:00.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7f5e5b87d640 1 -- 172.21.15.77:0/2987649598 wait complete. 2024-04-25T01:24:00.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3950125420 >> 172.21.15.77:0/3950125420 conn(0x7fe5880b1820 msgr2=0x7fe5880b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:00.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fde9a790640 1 -- 172.21.15.77:0/135022163 shutdown_connections 2024-04-25T01:24:00.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fde9a790640 1 -- 172.21.15.77:0/135022163 wait complete. 2024-04-25T01:24:00.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7feb6db6a640 1 --2- 172.21.15.77:0/358850826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb6815f630 0x7feb680ae9f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7feb680720d0 tx=0x7feb5802e730 comp rx=0 tx=0).stop 2024-04-25T01:24:00.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7feb6db6a640 1 -- 172.21.15.77:0/358850826 shutdown_connections 2024-04-25T01:24:00.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7feb6db6a640 1 --2- 172.21.15.77:0/358850826 >> v2:172.21.15.121:3300/0 conn(0x7feb680d28a0 0x7feb680b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7feb6db6a640 1 --2- 172.21.15.77:0/358850826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb6815f630 0x7feb680ae9f0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7feb6db6a640 1 -- 172.21.15.77:0/358850826 >> 172.21.15.77:0/358850826 conn(0x7feb680d1ca0 msgr2=0x7feb680d20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:00.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3950125420 shutdown_connections 2024-04-25T01:24:00.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7feb6db6a640 1 -- 172.21.15.77:0/358850826 shutdown_connections 2024-04-25T01:24:00.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3950125420 wait complete. 2024-04-25T01:24:00.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.129+0000 7feb6db6a640 1 -- 172.21.15.77:0/358850826 wait complete. 2024-04-25T01:24:00.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.130+0000 7feb6db6a640 1 Processor -- start 2024-04-25T01:24:00.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.130+0000 7f5e5b87d640 1 Processor -- start 2024-04-25T01:24:00.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.130+0000 7fde9a790640 1 Processor -- start 2024-04-25T01:24:00.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.130+0000 7f5e5b87d640 1 -- start start 2024-04-25T01:24:00.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.130+0000 7feb6db6a640 1 -- start start 2024-04-25T01:24:00.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7fe58fb5a640 1 Processor -- start 2024-04-25T01:24:00.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7fde9a790640 1 -- start start 2024-04-25T01:24:00.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7fe58fb5a640 1 -- start start 2024-04-25T01:24:00.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7feb6db6a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb680b0920 0x7feb680b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7feb6db6a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb680d28a0 0x7feb6815a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7f5e5b87d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54143940 0x7f5e54142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7f5e5b87d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e54143d20 0x7f5e541f1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7feb6db6a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb6815f630 0x7feb6815a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7f5e5b87d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e5415f400 0x7f5e54158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7feb6db6a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feb680aef30 con 0x7feb6815f630 2024-04-25T01:24:00.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7feb6db6a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feb680b1460 con 0x7feb680d28a0 2024-04-25T01:24:00.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7feb6db6a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7feb680af0b0 con 0x7feb680b0920 2024-04-25T01:24:00.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7feb67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb6815f630 0x7feb6815a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7f5e5b87d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5406ef60 con 0x7f5e54143d20 2024-04-25T01:24:00.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7f5e5b87d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5406ede0 con 0x7f5e54143940 2024-04-25T01:24:00.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7f5e5b87d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5e54072330 con 0x7f5e5415f400 2024-04-25T01:24:00.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7f5e58e18640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e54143d20 0x7f5e541f1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7f5e58e18640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e54143d20 0x7f5e541f1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45406/0 (socket says 172.21.15.77:45406) 2024-04-25T01:24:00.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7f5e58e18640 1 -- 172.21.15.77:0/3846128098 learned_addr learned my addr 172.21.15.77:0/3846128098 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:00.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7feb67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb6815f630 0x7feb6815a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45398/0 (socket says 172.21.15.77:45398) 2024-04-25T01:24:00.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7feb67fff640 1 -- 172.21.15.77:0/2759776216 learned_addr learned my addr 172.21.15.77:0/2759776216 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:00.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7feb677fe640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb680b0920 0x7feb680b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7f5e59619640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54143940 0x7f5e54142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7f5e59e1a640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e5415f400 0x7f5e54158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7fe58fb5a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe588143940 0x7fe58815c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.131+0000 7fe58fb5a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe588143d20 0x7fe588152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58fb5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58815f400 0x7fe58815ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7feb66ffd640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb680d28a0 0x7feb6815a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58d0f5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe588143d20 0x7fe588152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58fb5a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe58806d8d0 con 0x7fe58815f400 2024-04-25T01:24:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58fb5a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe58806d750 con 0x7fe588143d20 2024-04-25T01:24:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58fb5a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe58806da50 con 0x7fe588143940 2024-04-25T01:24:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58d8f6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe588143940 0x7fe58815c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58d0f5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe588143d20 0x7fe588152850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:34680/0 (socket says 172.21.15.77:34680) 2024-04-25T01:24:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58d0f5640 1 -- 172.21.15.77:0/3007880915 learned_addr learned my addr 172.21.15.77:0/3007880915 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58e0f7640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58815f400 0x7fe58815ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde9a790640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde940d2d40 0x7fde940b9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde9a790640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde940d4a10 0x7fde940b95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde9a790640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde9415e4c0 0x7fde940b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7feb67fff640 1 -- 172.21.15.77:0/2759776216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb680b0920 msgr2=0x7feb680b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7feb67fff640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb680b0920 0x7feb680b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7feb67fff640 1 -- 172.21.15.77:0/2759776216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb680d28a0 msgr2=0x7feb6815a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7feb67fff640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb680d28a0 0x7feb6815a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7feb67fff640 1 -- 172.21.15.77:0/2759776216 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6815aad0 con 0x7feb6815f630 2024-04-25T01:24:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde9a790640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fde9415ea20 con 0x7fde9415e4c0 2024-04-25T01:24:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde9a790640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fde9415e8a0 con 0x7fde940d4a10 2024-04-25T01:24:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde9a790640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fde940b45f0 con 0x7fde940d2d40 2024-04-25T01:24:00.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde93fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde940d2d40 0x7fde940b9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7feb677fe640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb680b0920 0x7feb680b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:00.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde93fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde940d2d40 0x7fde940b9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:56836/0 (socket says 172.21.15.77:56836) 2024-04-25T01:24:00.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde93fff640 1 -- 172.21.15.77:0/379325930 learned_addr learned my addr 172.21.15.77:0/379325930 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:00.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7f5e59619640 1 -- 172.21.15.77:0/3846128098 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e5415f400 msgr2=0x7f5e54158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7f5e59619640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e5415f400 0x7f5e54158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7f5e59619640 1 -- 172.21.15.77:0/3846128098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e54143d20 msgr2=0x7f5e541f1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7f5e59619640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e54143d20 0x7f5e541f1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7f5e59619640 1 -- 172.21.15.77:0/3846128098 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e540a9600 con 0x7f5e54143940 2024-04-25T01:24:00.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde98d2d640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde9415e4c0 0x7fde940b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde937fe640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde940d4a10 0x7fde940b95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7feb67fff640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb6815f630 0x7feb6815a590 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7feb6000c740 tx=0x7feb6000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58e0f7640 1 -- 172.21.15.77:0/3007880915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe588143940 msgr2=0x7fe58815c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7f5e59619640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54143940 0x7f5e54142480 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5e50031000 tx=0x7f5e500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58e0f7640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe588143940 0x7fe58815c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58e0f7640 1 -- 172.21.15.77:0/3007880915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe588143d20 msgr2=0x7fe588152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58e0f7640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe588143d20 0x7fe588152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fe58e0f7640 1 -- 172.21.15.77:0/3007880915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe588152d90 con 0x7fe58815f400 2024-04-25T01:24:00.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde98d2d640 1 -- 172.21.15.77:0/379325930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde940d2d40 msgr2=0x7fde940b9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde98d2d640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde940d2d40 0x7fde940b9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde98d2d640 1 -- 172.21.15.77:0/379325930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde940d4a10 msgr2=0x7fde940b95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde98d2d640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde940d4a10 0x7fde940b95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde98d2d640 1 -- 172.21.15.77:0/379325930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde940b7790 con 0x7fde9415e4c0 2024-04-25T01:24:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.132+0000 7fde93fff640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde940d2d40 0x7fde940b9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7feb4ffff640 1 -- 172.21.15.77:0/2759776216 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60016020 con 0x7feb6815f630 2024-04-25T01:24:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7feb4ffff640 1 -- 172.21.15.77:0/2759776216 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb6000ce50 con 0x7feb6815f630 2024-04-25T01:24:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7feb6db6a640 1 -- 172.21.15.77:0/2759776216 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb680b4980 con 0x7feb6815f630 2024-04-25T01:24:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7fe58e0f7640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58815f400 0x7fe58815ae90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe58000eba0 tx=0x7fe58000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7f5e49ffb640 1 -- 172.21.15.77:0/3846128098 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50037040 con 0x7f5e54143940 2024-04-25T01:24:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7f5e49ffb640 1 -- 172.21.15.77:0/3846128098 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e50003cc0 con 0x7f5e54143940 2024-04-25T01:24:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7f5e49ffb640 1 -- 172.21.15.77:0/3846128098 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50032810 con 0x7f5e54143940 2024-04-25T01:24:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7feb6db6a640 1 -- 172.21.15.77:0/2759776216 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb680b4f40 con 0x7feb6815f630 2024-04-25T01:24:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7feb4ffff640 1 -- 172.21.15.77:0/2759776216 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60003970 con 0x7feb6815f630 2024-04-25T01:24:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7f5e5b87d640 1 -- 172.21.15.77:0/3846128098 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e541429c0 con 0x7f5e54143940 2024-04-25T01:24:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7fde98d2d640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde9415e4c0 0x7fde940b6b10 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fde94067e80 tx=0x7fde8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7fe57e7fc640 1 -- 172.21.15.77:0/3007880915 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe580018020 con 0x7fe58815f400 2024-04-25T01:24:00.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7fe57e7fc640 1 -- 172.21.15.77:0/3007880915 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe58000cd20 con 0x7fe58815f400 2024-04-25T01:24:00.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3007880915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58815b3d0 con 0x7fe58815f400 2024-04-25T01:24:00.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3007880915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe58815d100 con 0x7fe58815f400 2024-04-25T01:24:00.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.133+0000 7f5e5b87d640 1 -- 172.21.15.77:0/3846128098 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e540a9b80 con 0x7f5e54143940 2024-04-25T01:24:00.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.134+0000 7fe57e7fc640 1 -- 172.21.15.77:0/3007880915 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5800146c0 con 0x7fe58815f400 2024-04-25T01:24:00.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.134+0000 7fde90ff9640 1 -- 172.21.15.77:0/379325930 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c040070 con 0x7fde9415e4c0 2024-04-25T01:24:00.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.134+0000 7fde90ff9640 1 -- 172.21.15.77:0/379325930 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde8c030a60 con 0x7fde9415e4c0 2024-04-25T01:24:00.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.134+0000 7fde90ff9640 1 -- 172.21.15.77:0/379325930 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c030e50 con 0x7fde9415e4c0 2024-04-25T01:24:00.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.134+0000 7fde9a790640 1 -- 172.21.15.77:0/379325930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde940b7a70 con 0x7fde9415e4c0 2024-04-25T01:24:00.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.134+0000 7feb6db6a640 1 -- 172.21.15.77:0/2759776216 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb68068fb0 con 0x7feb6815f630 2024-04-25T01:24:00.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.134+0000 7fde9a790640 1 -- 172.21.15.77:0/379325930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde940b8030 con 0x7fde9415e4c0 2024-04-25T01:24:00.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.135+0000 7f5e5b87d640 1 -- 172.21.15.77:0/3846128098 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e54068fb0 con 0x7f5e54143940 2024-04-25T01:24:00.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.135+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3007880915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe570005290 con 0x7fe58815f400 2024-04-25T01:24:00.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.135+0000 7f5e49ffb640 1 -- 172.21.15.77:0/3846128098 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5e50044020 con 0x7f5e54143940 2024-04-25T01:24:00.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.135+0000 7f5e49ffb640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e3806be30 0x7f5e3806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7f5e49ffb640 1 -- 172.21.15.77:0/3846128098 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f5e500aadf0 con 0x7f5e54143940 2024-04-25T01:24:00.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.135+0000 7feb4ffff640 1 -- 172.21.15.77:0/2759776216 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7feb60003b10 con 0x7feb6815f630 2024-04-25T01:24:00.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7feb4ffff640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feb5406be30 0x7feb5406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7feb4ffff640 1 -- 172.21.15.77:0/2759776216 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7feb6008abf0 con 0x7feb6815f630 2024-04-25T01:24:00.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7f5e58e18640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e3806be30 0x7f5e3806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7fe57e7fc640 1 -- 172.21.15.77:0/3007880915 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe580005d10 con 0x7fe58815f400 2024-04-25T01:24:00.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7fde9a790640 1 -- 172.21.15.77:0/379325930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde94068fb0 con 0x7fde9415e4c0 2024-04-25T01:24:00.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7feb677fe640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feb5406be30 0x7feb5406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7fe57e7fc640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56c06bf30 0x7fe56c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7fde90ff9640 1 -- 172.21.15.77:0/379325930 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fde8c039040 con 0x7fde9415e4c0 2024-04-25T01:24:00.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7fe57e7fc640 1 -- 172.21.15.77:0/3007880915 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fe58008bb10 con 0x7fe58815f400 2024-04-25T01:24:00.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7fe58d8f6640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56c06bf30 0x7fe56c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.136+0000 7f5e58e18640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e3806be30 0x7f5e3806e2f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5e540cbd10 tx=0x7f5e44009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.137+0000 7fde90ff9640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde8006c0d0 0x7fde8006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.137+0000 7feb677fe640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feb5406be30 0x7feb5406e2f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feb58037000 tx=0x7feb5800d000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.137+0000 7fe58d8f6640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56c06bf30 0x7fe56c06e3f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe584034000 tx=0x7fe58402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.137+0000 7fde90ff9640 1 -- 172.21.15.77:0/379325930 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fde8c0ac410 con 0x7fde9415e4c0 2024-04-25T01:24:00.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.137+0000 7fde93fff640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde8006c0d0 0x7fde8006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.137+0000 7fde93fff640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde8006c0d0 0x7fde8006e590 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fde84009f50 tx=0x7fde84009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.140+0000 7f5e49ffb640 1 -- 172.21.15.77:0/3846128098 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5e5004b020 con 0x7f5e54143940 2024-04-25T01:24:00.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.140+0000 7feb4ffff640 1 -- 172.21.15.77:0/2759776216 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7feb60020020 con 0x7feb6815f630 2024-04-25T01:24:00.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.145+0000 7fe57e7fc640 1 -- 172.21.15.77:0/3007880915 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe580022030 con 0x7fe58815f400 2024-04-25T01:24:00.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.149+0000 7fde90ff9640 1 -- 172.21.15.77:0/379325930 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fde8c04a060 con 0x7fde9415e4c0 2024-04-25T01:24:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.393+0000 7f5e5b87d640 1 -- 172.21.15.77:0/3846128098 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) -- 0x7f5e540cf240 con 0x7f5e54143940 2024-04-25T01:24:00.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.401+0000 7fde9a790640 1 -- 172.21.15.77:0/379325930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) -- 0x7fde940d42d0 con 0x7fde9415e4c0 2024-04-25T01:24:00.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.438+0000 7feb6db6a640 1 -- 172.21.15.77:0/2759776216 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) -- 0x7feb680cf180 con 0x7feb6815f630 2024-04-25T01:24:00.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.443+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3007880915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) -- 0x7fe570005580 con 0x7fe58815f400 2024-04-25T01:24:00.928 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-04-25T01:24:00.948+0000 7f46af24e640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:24:00.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.949+0000 7f5e49ffb640 1 -- 172.21.15.77:0/3846128098 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 5889 v29) ==== 95+0+0 (secure 0 0 0) 0x7f5e50079050 con 0x7f5e54143940 2024-04-25T01:24:00.929 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 5889 2024-04-25T01:24:00.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.949+0000 7fde90ff9640 1 -- 172.21.15.77:0/379325930 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5901 v30) ==== 95+0+0 (secure 0 0 0) 0x7fde8c07a650 con 0x7fde9415e4c0 2024-04-25T01:24:00.929 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 5901 2024-04-25T01:24:00.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.951+0000 7f5e5b87d640 1 -- 172.21.15.77:0/3846128098 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e3806be30 msgr2=0x7f5e3806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.951+0000 7f5e5b87d640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e3806be30 0x7f5e3806e2f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5e540cbd10 tx=0x7f5e44009040 comp rx=0 tx=0).stop 2024-04-25T01:24:00.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7f5e5b87d640 1 -- 172.21.15.77:0/3846128098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54143940 msgr2=0x7f5e54142480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7f5e5b87d640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54143940 0x7f5e54142480 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5e50031000 tx=0x7f5e500046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7f5e5b87d640 1 -- 172.21.15.77:0/3846128098 shutdown_connections 2024-04-25T01:24:00.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7f5e5b87d640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f5e3806be30 0x7f5e3806e2f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7f5e5b87d640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e54143d20 0x7f5e541f1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7f5e5b87d640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e5415f400 0x7f5e54158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7f5e5b87d640 1 --2- 172.21.15.77:0/3846128098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54143940 0x7f5e54142480 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7f5e5b87d640 1 -- 172.21.15.77:0/3846128098 >> 172.21.15.77:0/3846128098 conn(0x7f5e540b1820 msgr2=0x7f5e5415e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:00.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7f5e5b87d640 1 -- 172.21.15.77:0/3846128098 shutdown_connections 2024-04-25T01:24:00.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7f5e5b87d640 1 -- 172.21.15.77:0/3846128098 wait complete. 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7fde9a790640 1 -- 172.21.15.77:0/379325930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde8006c0d0 msgr2=0x7fde8006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7fde9a790640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde8006c0d0 0x7fde8006e590 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fde84009f50 tx=0x7fde84009190 comp rx=0 tx=0).stop 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7fde9a790640 1 -- 172.21.15.77:0/379325930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde9415e4c0 msgr2=0x7fde940b6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.952+0000 7fde9a790640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde9415e4c0 0x7fde940b6b10 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fde94067e80 tx=0x7fde8c004a40 comp rx=0 tx=0).stop 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.953+0000 7fde9a790640 1 -- 172.21.15.77:0/379325930 shutdown_connections 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.953+0000 7fde9a790640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fde8006c0d0 0x7fde8006e590 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.953+0000 7fde9a790640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde9415e4c0 0x7fde940b6b10 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.953+0000 7fde9a790640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde940d4a10 0x7fde940b95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.953+0000 7fde9a790640 1 --2- 172.21.15.77:0/379325930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde940d2d40 0x7fde940b9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.953+0000 7fde9a790640 1 -- 172.21.15.77:0/379325930 >> 172.21.15.77:0/379325930 conn(0x7fde940d3ef0 msgr2=0x7fde9406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.953+0000 7fde9a790640 1 -- 172.21.15.77:0/379325930 shutdown_connections 2024-04-25T01:24:00.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.953+0000 7fde9a790640 1 -- 172.21.15.77:0/379325930 wait complete. 2024-04-25T01:24:00.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.954+0000 7feb4ffff640 1 -- 172.21.15.77:0/2759776216 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5254 v31) ==== 95+0+0 (secure 0 0 0) 0x7feb6815aad0 con 0x7feb6815f630 2024-04-25T01:24:00.935 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 5254 2024-04-25T01:24:00.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 -- 172.21.15.77:0/2759776216 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feb5406be30 msgr2=0x7feb5406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feb5406be30 0x7feb5406e2f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feb58037000 tx=0x7feb5800d000 comp rx=0 tx=0).stop 2024-04-25T01:24:00.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 -- 172.21.15.77:0/2759776216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb6815f630 msgr2=0x7feb6815a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb6815f630 0x7feb6815a590 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7feb6000c740 tx=0x7feb6000cc00 comp rx=0 tx=0).stop 2024-04-25T01:24:00.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 -- 172.21.15.77:0/2759776216 shutdown_connections 2024-04-25T01:24:00.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7feb5406be30 0x7feb5406e2f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb6815f630 0x7feb6815a590 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb680d28a0 0x7feb6815a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 --2- 172.21.15.77:0/2759776216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb680b0920 0x7feb680b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 -- 172.21.15.77:0/2759776216 >> 172.21.15.77:0/2759776216 conn(0x7feb680d1ca0 msgr2=0x7feb6806eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:00.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 -- 172.21.15.77:0/2759776216 shutdown_connections 2024-04-25T01:24:00.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:00.957+0000 7feb6db6a640 1 -- 172.21.15.77:0/2759776216 wait complete. 2024-04-25T01:24:00.946 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T01:24:00.946 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T01:24:00.948 INFO:tasks.ceph.mds.a:Started 2024-04-25T01:24:00.948 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:00.951 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T01:24:00.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T01:24:00.953 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T01:24:00.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T01:24:00.954 INFO:tasks.ceph.mds.b:Started 2024-04-25T01:24:00.955 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:00.958 INFO:tasks.ceph.mds.d:Started 2024-04-25T01:24:00.959 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:01.022 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-04-25T01:24:01.032 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-25T01:24:01.033 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-25T01:24:01.927 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-04-25T01:24:01.947+0000 7f46af24e640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:24:01.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.950+0000 7fe57e7fc640 1 -- 172.21.15.77:0/3007880915 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 5886 v32) ==== 95+0+0 (secure 0 0 0) 0x7fe588152d90 con 0x7fe58815f400 2024-04-25T01:24:01.930 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 5886 2024-04-25T01:24:01.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.952+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3007880915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56c06bf30 msgr2=0x7fe56c06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:01.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.952+0000 7fe58fb5a640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56c06bf30 0x7fe56c06e3f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe584034000 tx=0x7fe58402e000 comp rx=0 tx=0).stop 2024-04-25T01:24:01.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.952+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3007880915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58815f400 msgr2=0x7fe58815ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:01.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.952+0000 7fe58fb5a640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58815f400 0x7fe58815ae90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe58000eba0 tx=0x7fe58000c490 comp rx=0 tx=0).stop 2024-04-25T01:24:01.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.952+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3007880915 shutdown_connections 2024-04-25T01:24:01.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.952+0000 7fe58fb5a640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fe56c06bf30 0x7fe56c06e3f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.952+0000 7fe58fb5a640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58815f400 0x7fe58815ae90 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.952+0000 7fe58fb5a640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe588143d20 0x7fe588152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.952+0000 7fe58fb5a640 1 --2- 172.21.15.77:0/3007880915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe588143940 0x7fe58815c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.952+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3007880915 >> 172.21.15.77:0/3007880915 conn(0x7fe5880b1820 msgr2=0x7fe5880b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:01.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.953+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3007880915 shutdown_connections 2024-04-25T01:24:01.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:01.953+0000 7fe58fb5a640 1 -- 172.21.15.77:0/3007880915 wait complete. 2024-04-25T01:24:01.948 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T01:24:01.949 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T01:24:01.951 INFO:tasks.ceph.mds.c:Started 2024-04-25T01:24:01.951 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:01.952 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:02.026 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-04-25T01:24:02.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.073+0000 7fddf3716640 1 Processor -- start 2024-04-25T01:24:02.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.073+0000 7fddf3716640 1 -- start start 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.074+0000 7fddf3716640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0cf650 0x7fddec0d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.074+0000 7fddf3716640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddec0cfbc0 con 0x7fddec161730 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.074+0000 7fddf3716640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddec0cfea0 con 0x7fddec0d0ba0 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.074+0000 7fddf3716640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fddec0d1010 con 0x7fddec0cf650 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.074+0000 7fddf14b2640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fddec161730 0x7fddec0d1200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41162/0 (socket says 172.21.15.77:41162) 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.074+0000 7fddf14b2640 1 -- 172.21.15.77:0/3656573576 learned_addr learned my addr 172.21.15.77:0/3656573576 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.074+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2322495983 0 0) 0x7fddec0cfbc0 con 0x7fddec161730 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.074+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddd8003520 con 0x7fddec161730 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.074+0000 7fddf0cb1640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0cf650 0x7fddec0d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1665695889 0 0) 0x7fddd8003520 con 0x7fddec161730 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0cf650 msgr2=0x7fddec0d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0cf650 0x7fddec0d0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 >> v1:172.21.15.121:6789/0 conn(0x7fddec0d0ba0 legacy=0x7fddec0b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec0b34f0 con 0x7fddec161730 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fdde8001670 con 0x7fddec161730 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 >> v1:172.21.15.77:6789/0 conn(0x7fddec161730 legacy=0x7fddec0d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddd80039e0 0x7fddd8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0cf650 0x7fddd8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddd8004d40 0x7fddd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fddec0d1010 con 0x7fddd80039e0 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fddd800ba70 con 0x7fddd8004d40 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fddd800bc70 con 0x7fddec0cf650 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fddf14b2640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddd80039e0 0x7fddd8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fddf0cb1640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0cf650 0x7fddd8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fddf1cb3640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddd8004d40 0x7fddd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fddf14b2640 1 -- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0cf650 msgr2=0x7fddd8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fddf14b2640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0cf650 0x7fddd8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fddf14b2640 1 -- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddd8004d40 msgr2=0x7fddd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fddf14b2640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddd8004d40 0x7fddd80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fddf14b2640 1 -- 172.21.15.77:0/3656573576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd8003520 con 0x7fddd80039e0 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.075+0000 7fddf1cb3640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddd8004d40 0x7fddd80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.076+0000 7fddf14b2640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddd80039e0 0x7fddd8003dc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fdde800c290 tx=0x7fddd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922e3fcc6c06781a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.076+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8007d50 con 0x7fddd80039e0 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.076+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde8005af0 con 0x7fddd80039e0 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.076+0000 7fdde3fff640 1 -- 172.21.15.77:0/3656573576 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8007070 con 0x7fddd80039e0 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.076+0000 7fddf3716640 1 -- 172.21.15.77:0/3656573576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddd80039e0 msgr2=0x7fddd8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.076+0000 7fddf3716640 1 --2- 172.21.15.77:0/3656573576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddd80039e0 0x7fddd8003dc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fdde800c290 tx=0x7fddd8005830 comp rx=0 tx=0).stop 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.076+0000 7fddf1cb3640 1 -- 172.21.15.77:0/3656573576 reap_dead start 2024-04-25T01:24:02.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.076+0000 7fddf3716640 1 -- 172.21.15.77:0/3656573576 shutdown_connections 2024-04-25T01:24:02.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.076+0000 7fddf3716640 1 -- 172.21.15.77:0/3656573576 >> 172.21.15.77:0/3656573576 conn(0x7fddec0d3f30 msgr2=0x7fddec0d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:02.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.076+0000 7fddf3716640 1 -- 172.21.15.77:0/3656573576 shutdown_connections 2024-04-25T01:24:02.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.076+0000 7fddf3716640 1 -- 172.21.15.77:0/3656573576 wait complete. 2024-04-25T01:24:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.078+0000 7fddf3716640 1 Processor -- start 2024-04-25T01:24:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.078+0000 7fddf3716640 1 -- start start 2024-04-25T01:24:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.078+0000 7fddf3716640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddd80039e0 0x7fddec0cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.078+0000 7fddf3716640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0b6e30 0x7fddec1f3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.078+0000 7fddf3716640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddec1f39a0 0x7fddec1f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.078+0000 7fddf3716640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fddec161b10 con 0x7fddec1f39a0 2024-04-25T01:24:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.078+0000 7fddf3716640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fddec0b3ab0 con 0x7fddd80039e0 2024-04-25T01:24:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.078+0000 7fddf3716640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fddec161c90 con 0x7fddec0b6e30 2024-04-25T01:24:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.078+0000 7fddf1cb3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddec1f39a0 0x7fddec1f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.078+0000 7fddf1cb3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddec1f39a0 0x7fddec1f3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45482/0 (socket says 172.21.15.77:45482) 2024-04-25T01:24:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf1cb3640 1 -- 172.21.15.77:0/1893124696 learned_addr learned my addr 172.21.15.77:0/1893124696 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf14b2640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddd80039e0 0x7fddec0cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf0cb1640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0b6e30 0x7fddec1f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf1cb3640 1 -- 172.21.15.77:0/1893124696 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0b6e30 msgr2=0x7fddec1f3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf1cb3640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0b6e30 0x7fddec1f3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf1cb3640 1 -- 172.21.15.77:0/1893124696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddd80039e0 msgr2=0x7fddec0cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf1cb3640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddd80039e0 0x7fddec0cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf1cb3640 1 -- 172.21.15.77:0/1893124696 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec0b93c0 con 0x7fddec1f39a0 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf0cb1640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0b6e30 0x7fddec1f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf1cb3640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddec1f39a0 0x7fddec1f3e30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fddec067db0 tx=0x7fdde400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fdde1ffb640 1 -- 172.21.15.77:0/1893124696 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde400ee50 con 0x7fddec1f39a0 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fdde1ffb640 1 -- 172.21.15.77:0/1893124696 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde4018070 con 0x7fddec1f39a0 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fdde1ffb640 1 -- 172.21.15.77:0/1893124696 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde4003dc0 con 0x7fddec1f39a0 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf3716640 1 -- 172.21.15.77:0/1893124696 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddec0b96a0 con 0x7fddec1f39a0 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.079+0000 7fddf3716640 1 -- 172.21.15.77:0/1893124696 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddec0b9c60 con 0x7fddec1f39a0 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.080+0000 7fddf3716640 1 -- 172.21.15.77:0/1893124696 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddec0b9150 con 0x7fddec1f39a0 2024-04-25T01:24:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.080+0000 7fdde1ffb640 1 -- 172.21.15.77:0/1893124696 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdde40050d0 con 0x7fddec1f39a0 2024-04-25T01:24:02.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.081+0000 7fdde1ffb640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fddd006be30 0x7fddd006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.081+0000 7fdde1ffb640 1 -- 172.21.15.77:0/1893124696 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7fdde408ae40 con 0x7fddec1f39a0 2024-04-25T01:24:02.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.081+0000 7fddf14b2640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fddd006be30 0x7fddd006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.081+0000 7fddf14b2640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fddd006be30 0x7fddd006e2f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdde8033000 tx=0x7fdde802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:02.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.085+0000 7fdde1ffb640 1 -- 172.21.15.77:0/1893124696 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdde40595b0 con 0x7fddec1f39a0 2024-04-25T01:24:02.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.295+0000 7fddf3716640 1 -- 172.21.15.77:0/1893124696 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddec068fb0 con 0x7fddec1f39a0 2024-04-25T01:24:02.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.296+0000 7fdde1ffb640 1 -- 172.21.15.77:0/1893124696 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+4356 (secure 0 0 0) 0x7fdde401be40 con 0x7fddec1f39a0 2024-04-25T01:24:02.276 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:02.276 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5129,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1374960539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1374960539},{"type":"v1","addr":"172.21.15.121:6834","nonce":1374960539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5467,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3910406676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":3910406676},{"type":"v1","addr":"172.21.15.121:6835","nonce":3910406676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:23:28.170422+0000","modified":"2024-04-25T01:24:01.956206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6835/2057169374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2057169374},{"type":"v1","addr":"172.21.15.77:6835","nonce":2057169374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-25T01:24:02.276 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-04-25T01:24:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 -- 172.21.15.77:0/1893124696 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fddd006be30 msgr2=0x7fddd006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fddd006be30 0x7fddd006e2f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdde8033000 tx=0x7fdde802e000 comp rx=0 tx=0).stop 2024-04-25T01:24:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 -- 172.21.15.77:0/1893124696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddec1f39a0 msgr2=0x7fddec1f3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddec1f39a0 0x7fddec1f3e30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fddec067db0 tx=0x7fdde400ec00 comp rx=0 tx=0).stop 2024-04-25T01:24:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 -- 172.21.15.77:0/1893124696 shutdown_connections 2024-04-25T01:24:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fddd006be30 0x7fddd006e2f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddec1f39a0 0x7fddec1f3e30 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddec0b6e30 0x7fddec1f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 --2- 172.21.15.77:0/1893124696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddd80039e0 0x7fddec0cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 -- 172.21.15.77:0/1893124696 >> 172.21.15.77:0/1893124696 conn(0x7fddec0d3f30 msgr2=0x7fddec15fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 -- 172.21.15.77:0/1893124696 shutdown_connections 2024-04-25T01:24:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:02.298+0000 7fddf3716640 1 -- 172.21.15.77:0/1893124696 wait complete. 2024-04-25T01:24:02.691 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:23:28.170422+0000', 'modified': '2024-04-25T01:24:01.956206+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6102}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6102': {'gid': 6102, 'name': 'a', 'rank': 0, 'incarnation': 33, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.77:6835/2057169374', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 2057169374}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 2057169374}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T01:24:02.691 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6102:up:replay 2024-04-25T01:24:03.692 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:03.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.820+0000 7fd2c3283640 1 Processor -- start 2024-04-25T01:24:03.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.821+0000 7fd2c3283640 1 -- start start 2024-04-25T01:24:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.821+0000 7fd2c3283640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd2bc154ac0 0x7fd2bc15b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.821+0000 7fd2c3283640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd2bc0cf2a0 0x7fd2bc0711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.821+0000 7fd2c3283640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc071730 0x7fd2bc06e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c3283640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fd2bc06d540 con 0x7fd2bc0cf2a0 2024-04-25T01:24:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c3283640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd2bc154ea0 con 0x7fd2bc154ac0 2024-04-25T01:24:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c3283640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2bc06d790 con 0x7fd2bc071730 2024-04-25T01:24:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c081e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd2bc0cf2a0 0x7fd2bc0711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c081e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd2bc0cf2a0 0x7fd2bc0711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45492/0 (socket says 172.21.15.77:45492) 2024-04-25T01:24:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c081e640 1 -- 172.21.15.77:0/1914987956 learned_addr learned my addr 172.21.15.77:0/1914987956 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c1820640 1 --2- 172.21.15.77:0/1914987956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc071730 0x7fd2bc06e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c101f640 1 --2- 172.21.15.77:0/1914987956 >> v2:172.21.15.121:3300/0 conn(0x7fd2bc154ac0 0x7fd2bc15b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c081e640 1 -- 172.21.15.77:0/1914987956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc071730 msgr2=0x7fd2bc06e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c081e640 1 --2- 172.21.15.77:0/1914987956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc071730 0x7fd2bc06e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c081e640 1 -- 172.21.15.77:0/1914987956 >> v2:172.21.15.121:3300/0 conn(0x7fd2bc154ac0 msgr2=0x7fd2bc15b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c081e640 1 --2- 172.21.15.77:0/1914987956 >> v2:172.21.15.121:3300/0 conn(0x7fd2bc154ac0 0x7fd2bc15b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.822+0000 7fd2c081e640 1 -- 172.21.15.77:0/1914987956 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2bc06da70 con 0x7fd2bc0cf2a0 2024-04-25T01:24:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.823+0000 7fd2c081e640 1 --2- 172.21.15.77:0/1914987956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2bc0cf2a0 0x7fd2bc0711f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0d0bc0 tx=0x7fd2b002b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d0386b1b10d105 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.823+0000 7fd2bbfff640 1 -- 172.21.15.77:0/1914987956 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b000b040 con 0x7fd2bc0cf2a0 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.823+0000 7fd2bbfff640 1 -- 172.21.15.77:0/1914987956 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b0030710 con 0x7fd2bc0cf2a0 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.823+0000 7fd2bbfff640 1 -- 172.21.15.77:0/1914987956 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0030ae0 con 0x7fd2bc0cf2a0 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.823+0000 7fd2c3283640 1 -- 172.21.15.77:0/1914987956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2bc0cf2a0 msgr2=0x7fd2bc0711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.823+0000 7fd2c3283640 1 --2- 172.21.15.77:0/1914987956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2bc0cf2a0 0x7fd2bc0711f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0d0bc0 tx=0x7fd2b002b5a0 comp rx=0 tx=0).stop 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.824+0000 7fd2c3283640 1 -- 172.21.15.77:0/1914987956 shutdown_connections 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.824+0000 7fd2c3283640 1 --2- 172.21.15.77:0/1914987956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc071730 0x7fd2bc06e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.824+0000 7fd2c3283640 1 --2- 172.21.15.77:0/1914987956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2bc0cf2a0 0x7fd2bc0711f0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.824+0000 7fd2c3283640 1 --2- 172.21.15.77:0/1914987956 >> v2:172.21.15.121:3300/0 conn(0x7fd2bc154ac0 0x7fd2bc15b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.824+0000 7fd2c3283640 1 -- 172.21.15.77:0/1914987956 >> 172.21.15.77:0/1914987956 conn(0x7fd2bc0b1820 msgr2=0x7fd2bc0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.824+0000 7fd2c3283640 1 -- 172.21.15.77:0/1914987956 shutdown_connections 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.824+0000 7fd2c3283640 1 -- 172.21.15.77:0/1914987956 wait complete. 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.825+0000 7fd2c3283640 1 Processor -- start 2024-04-25T01:24:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.825+0000 7fd2c3283640 1 -- start start 2024-04-25T01:24:03.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.825+0000 7fd2c3283640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc071730 0x7fd2bc14c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:03.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.825+0000 7fd2c3283640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2bc0cf2a0 0x7fd2bc1f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.825+0000 7fd2c3283640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2bc154ac0 0x7fd2bc158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c3283640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc0cf750 con 0x7fd2bc154ac0 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c3283640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc06e030 con 0x7fd2bc0cf2a0 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c3283640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2bc06ea90 con 0x7fd2bc071730 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c1820640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2bc154ac0 0x7fd2bc158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c1820640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2bc154ac0 0x7fd2bc158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45508/0 (socket says 172.21.15.77:45508) 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c1820640 1 -- 172.21.15.77:0/485700991 learned_addr learned my addr 172.21.15.77:0/485700991 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c101f640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc071730 0x7fd2bc14c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c081e640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2bc0cf2a0 0x7fd2bc1f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c1820640 1 -- 172.21.15.77:0/485700991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc071730 msgr2=0x7fd2bc14c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c1820640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc071730 0x7fd2bc14c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c1820640 1 -- 172.21.15.77:0/485700991 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2bc0cf2a0 msgr2=0x7fd2bc1f27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c1820640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2bc0cf2a0 0x7fd2bc1f27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c1820640 1 -- 172.21.15.77:0/485700991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2bc155380 con 0x7fd2bc154ac0 2024-04-25T01:24:03.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c081e640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2bc0cf2a0 0x7fd2bc1f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:03.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c1820640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2bc154ac0 0x7fd2bc158e40 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00e830 tx=0x7fd2ac00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:03.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2b9ffb640 1 -- 172.21.15.77:0/485700991 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac017070 con 0x7fd2bc154ac0 2024-04-25T01:24:03.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2b9ffb640 1 -- 172.21.15.77:0/485700991 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2ac00c940 con 0x7fd2bc154ac0 2024-04-25T01:24:03.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2b9ffb640 1 -- 172.21.15.77:0/485700991 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac00cd30 con 0x7fd2bc154ac0 2024-04-25T01:24:03.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.826+0000 7fd2c3283640 1 -- 172.21.15.77:0/485700991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2bc155df0 con 0x7fd2bc154ac0 2024-04-25T01:24:03.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.827+0000 7fd2c3283640 1 -- 172.21.15.77:0/485700991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2bc1563b0 con 0x7fd2bc154ac0 2024-04-25T01:24:03.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.827+0000 7fd2c3283640 1 -- 172.21.15.77:0/485700991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2bc068fb0 con 0x7fd2bc154ac0 2024-04-25T01:24:03.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.828+0000 7fd2b9ffb640 1 -- 172.21.15.77:0/485700991 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd2ac01a070 con 0x7fd2bc154ac0 2024-04-25T01:24:03.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.828+0000 7fd2b9ffb640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd2a806be60 0x7fd2a806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:03.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.828+0000 7fd2b9ffb640 1 -- 172.21.15.77:0/485700991 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7fd2ac08b030 con 0x7fd2bc154ac0 2024-04-25T01:24:03.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.828+0000 7fd2c101f640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd2a806be60 0x7fd2a806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:03.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.829+0000 7fd2c101f640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd2a806be60 0x7fd2a806e320 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4007380 tx=0x7fd2b400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:03.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:03.832+0000 7fd2b9ffb640 1 -- 172.21.15.77:0/485700991 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd2ac0597a0 con 0x7fd2bc154ac0 2024-04-25T01:24:04.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.043+0000 7fd2c3283640 1 -- 172.21.15.77:0/485700991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2bc151d70 con 0x7fd2bc154ac0 2024-04-25T01:24:04.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.044+0000 7fd2b9ffb640 1 -- 172.21.15.77:0/485700991 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5181 (secure 0 0 0) 0x7fd2ac0590a0 con 0x7fd2bc154ac0 2024-04-25T01:24:04.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:04.024 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5129,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1374960539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1374960539},{"type":"v1","addr":"172.21.15.121:6834","nonce":1374960539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5467,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3910406676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":3910406676},{"type":"v1","addr":"172.21.15.121:6835","nonce":3910406676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6120,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3088617511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3088617511},{"type":"v1","addr":"172.21.15.77:6837","nonce":3088617511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:23:28.170422+0000","modified":"2024-04-25T01:24:02.970582+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":33,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.77:6835/2057169374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2057169374},{"type":"v1","addr":"172.21.15.77:6835","nonce":2057169374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-25T01:24:04.024 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 35 2024-04-25T01:24:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 -- 172.21.15.77:0/485700991 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd2a806be60 msgr2=0x7fd2a806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd2a806be60 0x7fd2a806e320 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4007380 tx=0x7fd2b400b040 comp rx=0 tx=0).stop 2024-04-25T01:24:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 -- 172.21.15.77:0/485700991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2bc154ac0 msgr2=0x7fd2bc158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2bc154ac0 0x7fd2bc158e40 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00e830 tx=0x7fd2ac00ecf0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 -- 172.21.15.77:0/485700991 shutdown_connections 2024-04-25T01:24:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd2a806be60 0x7fd2a806e320 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2bc154ac0 0x7fd2bc158e40 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2bc0cf2a0 0x7fd2bc1f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 --2- 172.21.15.77:0/485700991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc071730 0x7fd2bc14c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 -- 172.21.15.77:0/485700991 >> 172.21.15.77:0/485700991 conn(0x7fd2bc0b1820 msgr2=0x7fd2bc14f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 -- 172.21.15.77:0/485700991 shutdown_connections 2024-04-25T01:24:04.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:04.046+0000 7fd2c3283640 1 -- 172.21.15.77:0/485700991 wait complete. 2024-04-25T01:24:04.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:23:28.170422+0000', 'modified': '2024-04-25T01:24:02.970582+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6102}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6102': {'gid': 6102, 'name': 'a', 'rank': 0, 'incarnation': 33, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.77:6835/2057169374', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 2057169374}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 2057169374}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T01:24:04.439 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6102:up:rejoin 2024-04-25T01:24:05.440 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:05.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.569+0000 7f3b76184640 1 Processor -- start 2024-04-25T01:24:05.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.569+0000 7f3b76184640 1 -- start start 2024-04-25T01:24:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b76184640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b70145590 0x7f3b700b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b76184640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b7015f9b0 0x7f3b7015fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b76184640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b700b4240 con 0x7f3b700ce110 2024-04-25T01:24:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b76184640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f3b70160830 con 0x7f3b7015f9b0 2024-04-25T01:24:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b76184640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3b70160350 con 0x7f3b70145590 2024-04-25T01:24:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b6effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b7015f9b0 0x7f3b7015fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b6effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b7015f9b0 0x7f3b7015fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45512/0 (socket says 172.21.15.77:45512) 2024-04-25T01:24:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b6effd640 1 -- 172.21.15.77:0/2466337813 learned_addr learned my addr 172.21.15.77:0/2466337813 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b6f7fe640 1 --2- 172.21.15.77:0/2466337813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b70145590 0x7f3b700b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b6effd640 1 -- 172.21.15.77:0/2466337813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b70145590 msgr2=0x7f3b700b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b6effd640 1 --2- 172.21.15.77:0/2466337813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b70145590 0x7f3b700b4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b6effd640 1 -- 172.21.15.77:0/2466337813 >> v1:172.21.15.121:6789/0 conn(0x7f3b700ce110 legacy=0x7f3b700ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.570+0000 7f3b6effd640 1 -- 172.21.15.77:0/2466337813 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b700b4f60 con 0x7f3b7015f9b0 2024-04-25T01:24:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.571+0000 7f3b6effd640 1 --2- 172.21.15.77:0/2466337813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7015f9b0 0x7f3b7015fe10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3b64009870 tx=0x7f3b6402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b497639cabbea72b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.571+0000 7f3b6e7fc640 1 -- 172.21.15.77:0/2466337813 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64037070 con 0x7f3b7015f9b0 2024-04-25T01:24:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.571+0000 7f3b6e7fc640 1 -- 172.21.15.77:0/2466337813 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b64032440 con 0x7f3b7015f9b0 2024-04-25T01:24:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.571+0000 7f3b6e7fc640 1 -- 172.21.15.77:0/2466337813 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64032810 con 0x7f3b7015f9b0 2024-04-25T01:24:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.572+0000 7f3b76184640 1 -- 172.21.15.77:0/2466337813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7015f9b0 msgr2=0x7f3b7015fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.572+0000 7f3b76184640 1 --2- 172.21.15.77:0/2466337813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7015f9b0 0x7f3b7015fe10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3b64009870 tx=0x7f3b6402a270 comp rx=0 tx=0).stop 2024-04-25T01:24:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.572+0000 7f3b76184640 1 -- 172.21.15.77:0/2466337813 shutdown_connections 2024-04-25T01:24:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.572+0000 7f3b76184640 1 --2- 172.21.15.77:0/2466337813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7015f9b0 0x7f3b7015fe10 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.572+0000 7f3b76184640 1 --2- 172.21.15.77:0/2466337813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b70145590 0x7f3b700b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.572+0000 7f3b76184640 1 -- 172.21.15.77:0/2466337813 >> 172.21.15.77:0/2466337813 conn(0x7f3b700b1820 msgr2=0x7f3b700b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:05.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.572+0000 7f3b76184640 1 -- 172.21.15.77:0/2466337813 shutdown_connections 2024-04-25T01:24:05.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.572+0000 7f3b76184640 1 -- 172.21.15.77:0/2466337813 wait complete. 2024-04-25T01:24:05.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.573+0000 7f3b76184640 1 Processor -- start 2024-04-25T01:24:05.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.573+0000 7f3b76184640 1 -- start start 2024-04-25T01:24:05.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.574+0000 7f3b76184640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b700ce110 0x7f3b701f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:05.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.574+0000 7f3b76184640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b70145590 0x7f3b7015d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:05.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.574+0000 7f3b76184640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7015f9b0 0x7f3b7013f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:05.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.574+0000 7f3b76184640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3b700b3c80 con 0x7f3b7015f9b0 2024-04-25T01:24:05.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.574+0000 7f3b76184640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7006cde0 con 0x7f3b700ce110 2024-04-25T01:24:05.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.574+0000 7f3b76184640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3b700b3f60 con 0x7f3b70145590 2024-04-25T01:24:05.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.574+0000 7f3b6ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7015f9b0 0x7f3b7013f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:05.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.574+0000 7f3b6ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7015f9b0 0x7f3b7013f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45528/0 (socket says 172.21.15.77:45528) 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.574+0000 7f3b6ffff640 1 -- 172.21.15.77:0/596153034 learned_addr learned my addr 172.21.15.77:0/596153034 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.574+0000 7f3b6effd640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b70145590 0x7f3b7015d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.574+0000 7f3b6f7fe640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b700ce110 0x7f3b701f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.575+0000 7f3b6ffff640 1 -- 172.21.15.77:0/596153034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b70145590 msgr2=0x7f3b7015d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.575+0000 7f3b6ffff640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b70145590 0x7f3b7015d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.575+0000 7f3b6ffff640 1 -- 172.21.15.77:0/596153034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b700ce110 msgr2=0x7f3b701f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.575+0000 7f3b6ffff640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b700ce110 0x7f3b701f4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.575+0000 7f3b6ffff640 1 -- 172.21.15.77:0/596153034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7013fc00 con 0x7f3b7015f9b0 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.575+0000 7f3b6ffff640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7015f9b0 0x7f3b7013f2c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f3b6800c710 tx=0x7f3b6800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.575+0000 7f3b57fff640 1 -- 172.21.15.77:0/596153034 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b68016020 con 0x7f3b7015f9b0 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.575+0000 7f3b57fff640 1 -- 172.21.15.77:0/596153034 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b68006d50 con 0x7f3b7015f9b0 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.575+0000 7f3b76184640 1 -- 172.21.15.77:0/596153034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b700adb60 con 0x7f3b7015f9b0 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.575+0000 7f3b57fff640 1 -- 172.21.15.77:0/596153034 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b68003bb0 con 0x7f3b7015f9b0 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.575+0000 7f3b76184640 1 -- 172.21.15.77:0/596153034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b7015db40 con 0x7f3b7015f9b0 2024-04-25T01:24:05.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.576+0000 7f3b76184640 1 -- 172.21.15.77:0/596153034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b70068fb0 con 0x7f3b7015f9b0 2024-04-25T01:24:05.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.576+0000 7f3b57fff640 1 -- 172.21.15.77:0/596153034 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3b6800a850 con 0x7f3b7015f9b0 2024-04-25T01:24:05.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.577+0000 7f3b57fff640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b5c06be30 0x7f3b5c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:05.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.577+0000 7f3b6f7fe640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b5c06be30 0x7f3b5c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:05.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.577+0000 7f3b57fff640 1 -- 172.21.15.77:0/596153034 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f3b6808ad00 con 0x7f3b7015f9b0 2024-04-25T01:24:05.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.577+0000 7f3b6f7fe640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b5c06be30 0x7f3b5c06e2f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3b60006fd0 tx=0x7f3b60008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:05.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.581+0000 7f3b57fff640 1 -- 172.21.15.77:0/596153034 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3b68020020 con 0x7f3b7015f9b0 2024-04-25T01:24:05.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.787+0000 7f3b76184640 1 -- 172.21.15.77:0/596153034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b70144d40 con 0x7f3b7015f9b0 2024-04-25T01:24:05.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.787+0000 7f3b57fff640 1 -- 172.21.15.77:0/596153034 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5188 (secure 0 0 0) 0x7f3b68019310 con 0x7f3b7015f9b0 2024-04-25T01:24:05.767 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:05.768 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5129,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1374960539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1374960539},{"type":"v1","addr":"172.21.15.121:6834","nonce":1374960539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5467,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3910406676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":3910406676},{"type":"v1","addr":"172.21.15.121:6835","nonce":3910406676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6120,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3088617511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3088617511},{"type":"v1","addr":"172.21.15.77:6837","nonce":3088617511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:23:28.170422+0000","modified":"2024-04-25T01:24:04.977155+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/2057169374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2057169374},{"type":"v1","addr":"172.21.15.77:6835","nonce":2057169374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6102,"qdb_cluster":[6102]},"id":3}]} 2024-04-25T01:24:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 36 2024-04-25T01:24:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 -- 172.21.15.77:0/596153034 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b5c06be30 msgr2=0x7f3b5c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b5c06be30 0x7f3b5c06e2f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3b60006fd0 tx=0x7f3b60008040 comp rx=0 tx=0).stop 2024-04-25T01:24:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 -- 172.21.15.77:0/596153034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7015f9b0 msgr2=0x7f3b7013f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7015f9b0 0x7f3b7013f2c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f3b6800c710 tx=0x7f3b6800cbd0 comp rx=0 tx=0).stop 2024-04-25T01:24:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 -- 172.21.15.77:0/596153034 shutdown_connections 2024-04-25T01:24:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f3b5c06be30 0x7f3b5c06e2f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7015f9b0 0x7f3b7013f2c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b70145590 0x7f3b7015d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 --2- 172.21.15.77:0/596153034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b700ce110 0x7f3b701f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 -- 172.21.15.77:0/596153034 >> 172.21.15.77:0/596153034 conn(0x7f3b700b1820 msgr2=0x7f3b700b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 -- 172.21.15.77:0/596153034 shutdown_connections 2024-04-25T01:24:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:05.790+0000 7f3b76184640 1 -- 172.21.15.77:0/596153034 wait complete. 2024-04-25T01:24:06.183 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:23:28.170422+0000', 'modified': '2024-04-25T01:24:04.977155+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6102}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6102': {'gid': 6102, 'name': 'a', 'rank': 0, 'incarnation': 33, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.77:6835/2057169374', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 2057169374}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 2057169374}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6102, 'qdb_cluster': [6102]} 2024-04-25T01:24:06.183 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:24:06.184 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T01:24:06.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.304+0000 7f5794e64640 1 Processor -- start 2024-04-25T01:24:06.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.304+0000 7f5794e64640 1 -- start start 2024-04-25T01:24:06.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.305+0000 7f5794e64640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f579015e4c0 0x7f57900d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.305+0000 7f5794e64640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f57900d2d40 0x7f57900cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.305+0000 7f5794e64640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57900d1e90 con 0x7f57900d4a10 2024-04-25T01:24:06.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.305+0000 7f5794e64640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f57900b54a0 con 0x7f57900d2d40 2024-04-25T01:24:06.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.305+0000 7f5794e64640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f57900d2130 con 0x7f579015e4c0 2024-04-25T01:24:06.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.305+0000 7f578ed9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f57900d2d40 0x7f57900cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.305+0000 7f578dd9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f57900d4a10 0x7f57900b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41170/0 (socket says 172.21.15.77:41170) 2024-04-25T01:24:06.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.305+0000 7f578ed9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f57900d2d40 0x7f57900cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45540/0 (socket says 172.21.15.77:45540) 2024-04-25T01:24:06.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.306+0000 7f578dd9b640 1 -- 172.21.15.77:0/2306381199 learned_addr learned my addr 172.21.15.77:0/2306381199 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:06.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.306+0000 7f578e59c640 1 --2- 172.21.15.77:0/2306381199 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f579015e4c0 0x7f57900d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.306+0000 7f578ed9d640 1 -- 172.21.15.77:0/2306381199 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f579015e4c0 msgr2=0x7f57900d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:06.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.306+0000 7f578ed9d640 1 --2- 172.21.15.77:0/2306381199 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f579015e4c0 0x7f57900d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.306+0000 7f578ed9d640 1 -- 172.21.15.77:0/2306381199 >> v1:172.21.15.77:6789/0 conn(0x7f57900d4a10 legacy=0x7f57900b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.306+0000 7f578ed9d640 1 -- 172.21.15.77:0/2306381199 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57900b5780 con 0x7f57900d2d40 2024-04-25T01:24:06.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.306+0000 7f578ed9d640 1 --2- 172.21.15.77:0/2306381199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57900d2d40 0x7f57900cfaf0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f57880097a0 tx=0x7f578802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d3a14842b5149a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.307+0000 7f578d59a640 1 -- 172.21.15.77:0/2306381199 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5788037070 con 0x7f57900d2d40 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.307+0000 7f578d59a640 1 -- 172.21.15.77:0/2306381199 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f578802ae50 con 0x7f57900d2d40 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.307+0000 7f578d59a640 1 -- 172.21.15.77:0/2306381199 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5788032660 con 0x7f57900d2d40 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.307+0000 7f5794e64640 1 -- 172.21.15.77:0/2306381199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57900d2d40 msgr2=0x7f57900cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.307+0000 7f5794e64640 1 --2- 172.21.15.77:0/2306381199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57900d2d40 0x7f57900cfaf0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f57880097a0 tx=0x7f578802a100 comp rx=0 tx=0).stop 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.307+0000 7f5794e64640 1 -- 172.21.15.77:0/2306381199 shutdown_connections 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.307+0000 7f5794e64640 1 --2- 172.21.15.77:0/2306381199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57900d2d40 0x7f57900cfaf0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.307+0000 7f5794e64640 1 --2- 172.21.15.77:0/2306381199 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f579015e4c0 0x7f57900d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.307+0000 7f5794e64640 1 -- 172.21.15.77:0/2306381199 >> 172.21.15.77:0/2306381199 conn(0x7f57900d3ef0 msgr2=0x7f57900d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.307+0000 7f5794e64640 1 -- 172.21.15.77:0/2306381199 shutdown_connections 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.307+0000 7f5794e64640 1 -- 172.21.15.77:0/2306381199 wait complete. 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.309+0000 7f5794e64640 1 Processor -- start 2024-04-25T01:24:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.309+0000 7f5794e64640 1 -- start start 2024-04-25T01:24:06.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.309+0000 7f5794e64640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57900d2d40 0x7f57901594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.309+0000 7f5794e64640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57900d4a10 0x7f57900c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.309+0000 7f5794e64640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f579015e4c0 0x7f57900b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.309+0000 7f5794e64640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f579015ea20 con 0x7f57900d4a10 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.309+0000 7f5794e64640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f579015e8a0 con 0x7f57900d2d40 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.309+0000 7f5794e64640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f57900b45f0 con 0x7f579015e4c0 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.309+0000 7f578dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57900d4a10 0x7f57900c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.309+0000 7f578dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57900d4a10 0x7f57900c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45554/0 (socket says 172.21.15.77:45554) 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.309+0000 7f578dd9b640 1 -- 172.21.15.77:0/851450501 learned_addr learned my addr 172.21.15.77:0/851450501 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f578ed9d640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f579015e4c0 0x7f57900b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f578e59c640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57900d2d40 0x7f57901594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f578dd9b640 1 -- 172.21.15.77:0/851450501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f579015e4c0 msgr2=0x7f57900b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f578dd9b640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f579015e4c0 0x7f57900b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f578dd9b640 1 -- 172.21.15.77:0/851450501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57900d2d40 msgr2=0x7f57901594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f578dd9b640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57900d2d40 0x7f57901594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f578dd9b640 1 -- 172.21.15.77:0/851450501 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57900cd680 con 0x7f57900d4a10 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f578e59c640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57900d2d40 0x7f57901594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f578dd9b640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57900d4a10 0x7f57900c9190 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f578400c470 tx=0x7f5784001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f577effd640 1 -- 172.21.15.77:0/851450501 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784017070 con 0x7f57900d4a10 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f577effd640 1 -- 172.21.15.77:0/851450501 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5784005ce0 con 0x7f57900d4a10 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f577effd640 1 -- 172.21.15.77:0/851450501 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57840072f0 con 0x7f57900d4a10 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f5794e64640 1 -- 172.21.15.77:0/851450501 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57900cd960 con 0x7f57900d4a10 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.310+0000 7f5794e64640 1 -- 172.21.15.77:0/851450501 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57900cb650 con 0x7f57900d4a10 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.311+0000 7f577effd640 1 -- 172.21.15.77:0/851450501 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f578401a070 con 0x7f57900d4a10 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.312+0000 7f577effd640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f577006be30 0x7f577006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.312+0000 7f578e59c640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f577006be30 0x7f577006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.312+0000 7f577effd640 1 -- 172.21.15.77:0/851450501 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f578408a4b0 con 0x7f57900d4a10 2024-04-25T01:24:06.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.312+0000 7f578e59c640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f577006be30 0x7f577006e2f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f57900b9cc0 tx=0x7f57800039f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.313+0000 7f5797b71740 1 Processor -- start 2024-04-25T01:24:06.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.313+0000 7f5797b71740 1 -- start start 2024-04-25T01:24:06.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f5797b71740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f5797b71740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb3142bc80 0x55fb3142b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f5797b71740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142b540 0x55fb31400350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f5797b71740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fb31363f00 con 0x55fb31448b60 2024-04-25T01:24:06.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f5797b71740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fb3130c750 con 0x55fb3142bc80 2024-04-25T01:24:06.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f5797b71740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55fb3135fec0 con 0x55fb3142b540 2024-04-25T01:24:06.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f578e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f578e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45562/0 (socket says 172.21.15.77:45562) 2024-04-25T01:24:06.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f578e59c640 1 -- 172.21.15.77:0/3181592996 learned_addr learned my addr 172.21.15.77:0/3181592996 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:06.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f578e59c640 1 -- 172.21.15.77:0/3181592996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142b540 msgr2=0x55fb31400350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:06.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f578e59c640 1 --2- 172.21.15.77:0/3181592996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142b540 0x55fb31400350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f578e59c640 1 -- 172.21.15.77:0/3181592996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb3142bc80 msgr2=0x55fb3142b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:06.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f578e59c640 1 --2- 172.21.15.77:0/3181592996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb3142bc80 0x55fb3142b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f578e59c640 1 -- 172.21.15.77:0/3181592996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb310782d0 con 0x55fb31448b60 2024-04-25T01:24:06.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f578e59c640 1 --2- 172.21.15.77:0/3181592996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fb4f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5780016470 tx=0x7f5780012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a48cb7f592d6df8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f576ffff640 1 -- 172.21.15.77:0/3181592996 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5780026070 con 0x55fb31448b60 2024-04-25T01:24:06.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f576ffff640 1 -- 172.21.15.77:0/3181592996 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57800148a0 con 0x55fb31448b60 2024-04-25T01:24:06.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.314+0000 7f576ffff640 1 -- 172.21.15.77:0/3181592996 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5780014c90 con 0x55fb31448b60 2024-04-25T01:24:06.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.315+0000 7f5797b71740 1 -- 172.21.15.77:0/3181592996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 msgr2=0x55fb313fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.315+0000 7f5797b71740 1 --2- 172.21.15.77:0/3181592996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fb4f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5780016470 tx=0x7f5780012000 comp rx=0 tx=0).stop 2024-04-25T01:24:06.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.315+0000 7f5797b71740 1 -- 172.21.15.77:0/3181592996 shutdown_connections 2024-04-25T01:24:06.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.315+0000 7f5797b71740 1 --2- 172.21.15.77:0/3181592996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142b540 0x55fb31400350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.315+0000 7f5797b71740 1 --2- 172.21.15.77:0/3181592996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb3142bc80 0x55fb3142b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.315+0000 7f5797b71740 1 --2- 172.21.15.77:0/3181592996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fb4f0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.315+0000 7f5797b71740 1 -- 172.21.15.77:0/3181592996 >> 172.21.15.77:0/3181592996 conn(0x55fb313fd760 msgr2=0x55fb31446000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.315+0000 7f5797b71740 1 -- 172.21.15.77:0/3181592996 shutdown_connections 2024-04-25T01:24:06.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.315+0000 7f5797b71740 1 -- 172.21.15.77:0/3181592996 wait complete. 2024-04-25T01:24:06.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.316+0000 7f5797b71740 1 -- start start 2024-04-25T01:24:06.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.316+0000 7f5797b71740 1 Processor -- start 2024-04-25T01:24:06.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f5797b71740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb3142bc80 0x55fb31457060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f5797b71740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fa650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f5797b71740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb31426580 0x55fb314269e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f5797b71740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fb313ff910 con 0x55fb31448b60 2024-04-25T01:24:06.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f5797b71740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fb3131fcf0 con 0x55fb3142bc80 2024-04-25T01:24:06.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f5797b71740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55fb313fed00 con 0x55fb31426580 2024-04-25T01:24:06.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f5797b71740 10 client.0 _ll_register_callbacks cb 0x55fb3131f6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:24:06.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fa650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fa650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45564/0 (socket says 172.21.15.77:45564) 2024-04-25T01:24:06.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578dd9b640 1 -- 172.21.15.77:0/4043672406 learned_addr learned my addr 172.21.15.77:0/4043672406 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:06.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578ed9d640 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb31426580 0x55fb314269e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578dd9b640 1 -- 172.21.15.77:0/4043672406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb31426580 msgr2=0x55fb314269e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:06.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578e59c640 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb3142bc80 0x55fb31457060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578dd9b640 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb31426580 0x55fb314269e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578dd9b640 1 -- 172.21.15.77:0/4043672406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb3142bc80 msgr2=0x55fb31457060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:06.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578dd9b640 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb3142bc80 0x55fb31457060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578dd9b640 1 -- 172.21.15.77:0/4043672406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb311426b0 con 0x55fb31448b60 2024-04-25T01:24:06.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578ed9d640 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb31426580 0x55fb314269e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:06.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578e59c640 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb3142bc80 0x55fb31457060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:06.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f578dd9b640 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fa650 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f578400cfd0 tx=0x7f5784058cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da80a240fc502b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f576d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:24:06.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f576d7fa640 1 -- 172.21.15.77:0/4043672406 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784070020 con 0x55fb31448b60 2024-04-25T01:24:06.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f576d7fa640 1 -- 172.21.15.77:0/4043672406 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f5784060e60 con 0x55fb31448b60 2024-04-25T01:24:06.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.317+0000 7f5797b71740 1 -- 172.21.15.77:0/4043672406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57800148a0 con 0x55fb31448b60 2024-04-25T01:24:06.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f576d7fa640 1 -- 172.21.15.77:0/4043672406 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f57840610f0 con 0x55fb31448b60 2024-04-25T01:24:06.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f5797b71740 10 client.6153 fetch_fsmap learned FSMap version 36 2024-04-25T01:24:06.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f5797b71740 1 -- 172.21.15.77:0/4043672406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x55fb3142e0e0 con 0x55fb31448b60 2024-04-25T01:24:06.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f576d7fa640 1 -- 172.21.15.77:0/4043672406 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5784084020 con 0x55fb31448b60 2024-04-25T01:24:06.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f576d7fa640 1 -- 172.21.15.77:0/4043672406 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f5784007490 con 0x55fb31448b60 2024-04-25T01:24:06.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f5797b71740 10 client.6153 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T01:24:06.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f5797b71740 10 client.6153 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T01:24:06.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f5797b71740 20 client.6153 populate_metadata read hostname 'smithi077' 2024-04-25T01:24:06.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f5797b71740 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.77:6834/2057169374,v1:172.21.15.77:6835/2057169374] conn(0x55fb3143d660 0x55fb3143da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f5797b71740 4 client.6153 mds_command: new command op to 6102 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:24:06.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f5797b71740 1 -- 172.21.15.77:0/4043672406 --> [v2:172.21.15.77:6834/2057169374,v1:172.21.15.77:6835/2057169374] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb31371440 con 0x55fb3143d660 2024-04-25T01:24:06.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f578e59c640 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.77:6834/2057169374,v1:172.21.15.77:6835/2057169374] conn(0x55fb3143d660 0x55fb3143da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f578e59c640 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.77:6834/2057169374,v1:172.21.15.77:6835/2057169374] conn(0x55fb3143d660 0x55fb3143da80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bccacfabdae7da90 server_cookie=b12ff831c3e21765 in_seq=0 out_seq=0 2024-04-25T01:24:06.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.318+0000 7f576d7fa640 10 client.6153 ms_handle_connect on v2:172.21.15.77:6834/2057169374 2024-04-25T01:24:06.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f576d7fa640 1 -- 172.21.15.77:0/4043672406 <== mds.0 v2:172.21.15.77:6834/2057169374 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb31371440 con 0x55fb3143d660 2024-04-25T01:24:06.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f576d7fa640 10 client.6153 handle_command_reply: tid=0 2024-04-25T01:24:06.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 1 client.6153 shutdown 2024-04-25T01:24:06.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 1 -- 172.21.15.77:0/4043672406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 msgr2=0x55fb313fa650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fa650 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f578400cfd0 tx=0x7f5784058cd0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 1 -- 172.21.15.77:0/4043672406 shutdown_connections 2024-04-25T01:24:06.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.77:6834/2057169374,v1:172.21.15.77:6835/2057169374] conn(0x55fb3143d660 0x55fb3143da80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31448b60 0x55fb313fa650 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb31426580 0x55fb314269e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 1 --2- 172.21.15.77:0/4043672406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb3142bc80 0x55fb31457060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f576d7fa640 0 client.6153 ms_handle_reset on v2:172.21.15.77:6834/2057169374 2024-04-25T01:24:06.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 1 -- 172.21.15.77:0/4043672406 >> 172.21.15.77:0/4043672406 conn(0x55fb3142b540 msgr2=0x55fb31446890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 1 -- 172.21.15.77:0/4043672406 shutdown_connections 2024-04-25T01:24:06.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 1 -- 172.21.15.77:0/4043672406 wait complete. 2024-04-25T01:24:06.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.319+0000 7f5797b71740 20 client.6153 trim_cache size 0 max 16384 2024-04-25T01:24:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.337+0000 7f5797b71740 1 Processor -- start 2024-04-25T01:24:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.337+0000 7f5797b71740 1 -- start start 2024-04-25T01:24:06.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f5797b71740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb314473e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f5797b71740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142bc80 0x55fb31400d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f5797b71740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb31448b60 0x55fb31401250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f5797b71740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f57901ffe90 con 0x55fb31426580 2024-04-25T01:24:06.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f5797b71740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f57901f7f50 con 0x55fb31448b60 2024-04-25T01:24:06.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f5797b71740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f57901ff840 con 0x55fb3142bc80 2024-04-25T01:24:06.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f578e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb314473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f578e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb314473e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45572/0 (socket says 172.21.15.77:45572) 2024-04-25T01:24:06.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f578e59c640 1 -- 172.21.15.77:0/4123124037 learned_addr learned my addr 172.21.15.77:0/4123124037 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f578ed9d640 1 --2- 172.21.15.77:0/4123124037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb31448b60 0x55fb31401250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f578dd9b640 1 --2- 172.21.15.77:0/4123124037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142bc80 0x55fb31400d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f578e59c640 1 -- 172.21.15.77:0/4123124037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142bc80 msgr2=0x55fb31400d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f578e59c640 1 --2- 172.21.15.77:0/4123124037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142bc80 0x55fb31400d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f578e59c640 1 -- 172.21.15.77:0/4123124037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb31448b60 msgr2=0x55fb31401250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f578e59c640 1 --2- 172.21.15.77:0/4123124037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb31448b60 0x55fb31401250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f578e59c640 1 -- 172.21.15.77:0/4123124037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb3143f1a0 con 0x55fb31426580 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.338+0000 7f578e59c640 1 --2- 172.21.15.77:0/4123124037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb314473e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5780004010 tx=0x7f5780014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2155bee683e3b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f576f7fe640 1 -- 172.21.15.77:0/4123124037 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5780038070 con 0x55fb31426580 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f576f7fe640 1 -- 172.21.15.77:0/4123124037 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5780032210 con 0x55fb31426580 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f576f7fe640 1 -- 172.21.15.77:0/4123124037 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5780032e60 con 0x55fb31426580 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f5797b71740 1 -- 172.21.15.77:0/4123124037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 msgr2=0x55fb314473e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f5797b71740 1 --2- 172.21.15.77:0/4123124037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb314473e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5780004010 tx=0x7f5780014000 comp rx=0 tx=0).stop 2024-04-25T01:24:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f5797b71740 1 -- 172.21.15.77:0/4123124037 shutdown_connections 2024-04-25T01:24:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f5797b71740 1 --2- 172.21.15.77:0/4123124037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb31448b60 0x55fb31401250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f5797b71740 1 --2- 172.21.15.77:0/4123124037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142bc80 0x55fb31400d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f5797b71740 1 --2- 172.21.15.77:0/4123124037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb314473e0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f5797b71740 1 -- 172.21.15.77:0/4123124037 >> 172.21.15.77:0/4123124037 conn(0x55fb3142b540 msgr2=0x55fb313fb2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f5797b71740 1 -- 172.21.15.77:0/4123124037 shutdown_connections 2024-04-25T01:24:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.339+0000 7f5797b71740 1 -- 172.21.15.77:0/4123124037 wait complete. 2024-04-25T01:24:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.340+0000 7f5797b71740 1 -- start start 2024-04-25T01:24:06.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.340+0000 7f5797b71740 1 Processor -- start 2024-04-25T01:24:06.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f5797b71740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb31400d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f5797b71740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142bc80 0x55fb31401250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f5797b71740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb31448b60 0x55fb314cc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f5797b71740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fb3142d2b0 con 0x55fb31426580 2024-04-25T01:24:06.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f5797b71740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fb31426f20 con 0x55fb31448b60 2024-04-25T01:24:06.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f5797b71740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55fb3143d110 con 0x55fb3142bc80 2024-04-25T01:24:06.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f5797b71740 10 client.0 _ll_register_callbacks cb 0x7f57840580e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:24:06.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb31400d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb31400d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45588/0 (socket says 172.21.15.77:45588) 2024-04-25T01:24:06.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578e59c640 1 -- 172.21.15.77:0/296854937 learned_addr learned my addr 172.21.15.77:0/296854937 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:06.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578dd9b640 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142bc80 0x55fb31401250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578ed9d640 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb31448b60 0x55fb314cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578e59c640 1 -- 172.21.15.77:0/296854937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142bc80 msgr2=0x55fb31401250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:06.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578e59c640 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142bc80 0x55fb31401250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578e59c640 1 -- 172.21.15.77:0/296854937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb31448b60 msgr2=0x55fb314cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:06.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578e59c640 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb31448b60 0x55fb314cc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578e59c640 1 -- 172.21.15.77:0/296854937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb31447560 con 0x55fb31426580 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578ed9d640 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb31448b60 0x55fb314cc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f578e59c640 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb31400d10 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5780016470 tx=0x7f5780016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7dcf465b726334 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f576d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f576d7fa640 1 -- 172.21.15.77:0/296854937 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5780038030 con 0x55fb31426580 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f576d7fa640 1 -- 172.21.15.77:0/296854937 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f5780014c60 con 0x55fb31426580 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f5797b71740 1 -- 172.21.15.77:0/296854937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5780032210 con 0x55fb31426580 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f576d7fa640 1 -- 172.21.15.77:0/296854937 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5780044050 con 0x55fb31426580 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f5797b71740 10 client.6159 fetch_fsmap learned FSMap version 36 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.341+0000 7f5797b71740 1 -- 172.21.15.77:0/296854937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x55fb3142e6d0 con 0x55fb31426580 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f576d7fa640 1 -- 172.21.15.77:0/296854937 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5780032210 con 0x55fb31426580 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f576d7fa640 1 -- 172.21.15.77:0/296854937 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f5780047020 con 0x55fb31426580 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f5797b71740 10 client.6159 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f5797b71740 10 client.6159 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f5797b71740 20 client.6159 populate_metadata read hostname 'smithi077' 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f5797b71740 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.77:6834/2057169374,v1:172.21.15.77:6835/2057169374] conn(0x55fb314cc880 0x55fb314ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f5797b71740 4 client.6159 mds_command: new command op to 6102 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f5797b71740 1 -- 172.21.15.77:0/296854937 --> [v2:172.21.15.77:6834/2057169374,v1:172.21.15.77:6835/2057169374] -- command(tid 0: {"prefix": "status"}) -- 0x55fb3136b790 con 0x55fb314cc880 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f578dd9b640 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.77:6834/2057169374,v1:172.21.15.77:6835/2057169374] conn(0x55fb314cc880 0x55fb314ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f576d7fa640 10 client.6159 ms_handle_connect on v2:172.21.15.77:6834/2057169374 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f578dd9b640 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.77:6834/2057169374,v1:172.21.15.77:6835/2057169374] conn(0x55fb314cc880 0x55fb314ccc60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b557d4105cec511 server_cookie=a0e9e9c4363f90e7 in_seq=0 out_seq=0 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f576d7fa640 1 -- 172.21.15.77:0/296854937 <== mds.0 v2:172.21.15.77:6834/2057169374 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55fb3136b790 con 0x55fb314cc880 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f576d7fa640 10 client.6159 handle_command_reply: tid=0 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f5797b71740 1 client.6159 shutdown 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f5797b71740 1 -- 172.21.15.77:0/296854937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 msgr2=0x55fb31400d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.342+0000 7f5797b71740 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb31400d10 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5780016470 tx=0x7f5780016000 comp rx=0 tx=0).stop 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5797b71740 1 -- 172.21.15.77:0/296854937 shutdown_connections 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5797b71740 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fb31448b60 0x55fb314cc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5797b71740 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fb3142bc80 0x55fb31401250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5797b71740 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.77:6834/2057169374,v1:172.21.15.77:6835/2057169374] conn(0x55fb314cc880 0x55fb314ccc60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5797b71740 1 --2- 172.21.15.77:0/296854937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fb31426580 0x55fb31400d10 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5797b71740 1 -- 172.21.15.77:0/296854937 >> 172.21.15.77:0/296854937 conn(0x55fb3142b540 msgr2=0x55fb31456c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f576d7fa640 0 client.6159 ms_handle_reset on v2:172.21.15.77:6834/2057169374 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5797b71740 1 -- 172.21.15.77:0/296854937 shutdown_connections 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5797b71740 1 -- 172.21.15.77:0/296854937 wait complete. 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5797b71740 20 client.6159 trim_cache size 0 max 16384 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "7c3c39ee-077e-476c-8994-eede4fab301d", 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 6102, 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 4.3824173899999996, 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 36, 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 55, 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 55, 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 5.301536757 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:24:06.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 -- 172.21.15.77:0/851450501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f577006be30 msgr2=0x7f577006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f577006be30 0x7f577006e2f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f57900b9cc0 tx=0x7f57800039f0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 -- 172.21.15.77:0/851450501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57900d4a10 msgr2=0x7f57900c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57900d4a10 0x7f57900c9190 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f578400c470 tx=0x7f5784001e70 comp rx=0 tx=0).stop 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 -- 172.21.15.77:0/851450501 shutdown_connections 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f577006be30 0x7f577006e2f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57900d4a10 0x7f57900c9190 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f579015e4c0 0x7f57900b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 --2- 172.21.15.77:0/851450501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57900d2d40 0x7f57901594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 -- 172.21.15.77:0/851450501 >> 172.21.15.77:0/851450501 conn(0x7f57900d3ef0 msgr2=0x7f579015d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 -- 172.21.15.77:0/851450501 shutdown_connections 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.343+0000 7f5794e64640 1 -- 172.21.15.77:0/851450501 wait complete. 2024-04-25T01:24:06.335 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T01:24:06.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.451+0000 7fd59e5a2640 1 Processor -- start 2024-04-25T01:24:06.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.451+0000 7fd59e5a2640 1 -- start start 2024-04-25T01:24:06.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.452+0000 7fd59e5a2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd5980b2ad0 0x7fd5980cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.452+0000 7fd59e5a2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd598162080 con 0x7fd598161b80 2024-04-25T01:24:06.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.452+0000 7fd59e5a2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd598162360 con 0x7fd598158700 2024-04-25T01:24:06.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.452+0000 7fd59e5a2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fd59806e840 con 0x7fd5980b2ad0 2024-04-25T01:24:06.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.452+0000 7fd5977fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd5980b2ad0 0x7fd5980cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.452+0000 7fd59cb3f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd598161b80 0x7fd59806d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41176/0 (socket says 172.21.15.77:41176) 2024-04-25T01:24:06.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.452+0000 7fd59cb3f640 1 -- 172.21.15.77:0/1410206137 learned_addr learned my addr 172.21.15.77:0/1410206137 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:06.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.452+0000 7fd5977fe640 1 -- 172.21.15.77:0/1410206137 >> v1:172.21.15.121:6789/0 conn(0x7fd598158700 legacy=0x7fd59815efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.452+0000 7fd5977fe640 1 -- 172.21.15.77:0/1410206137 >> v1:172.21.15.77:6789/0 conn(0x7fd598161b80 legacy=0x7fd59806d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.452+0000 7fd5977fe640 1 -- 172.21.15.77:0/1410206137 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd598162640 con 0x7fd5980b2ad0 2024-04-25T01:24:06.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.452+0000 7fd596ffd640 1 -- 172.21.15.77:0/1410206137 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1473402006 0 0) 0x7fd598162080 con 0x7fd598161b80 2024-04-25T01:24:06.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.453+0000 7fd5977fe640 1 --2- 172.21.15.77:0/1410206137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5980b2ad0 0x7fd5980cff00 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0097a0 tx=0x7fd58c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8766bfe8b7e9c923 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.453+0000 7fd596ffd640 1 -- 172.21.15.77:0/1410206137 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58c037070 con 0x7fd5980b2ad0 2024-04-25T01:24:06.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.453+0000 7fd596ffd640 1 -- 172.21.15.77:0/1410206137 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd58c032440 con 0x7fd5980b2ad0 2024-04-25T01:24:06.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.453+0000 7fd596ffd640 1 -- 172.21.15.77:0/1410206137 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58c032810 con 0x7fd5980b2ad0 2024-04-25T01:24:06.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.453+0000 7fd59e5a2640 1 -- 172.21.15.77:0/1410206137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5980b2ad0 msgr2=0x7fd5980cff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.453+0000 7fd59e5a2640 1 --2- 172.21.15.77:0/1410206137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5980b2ad0 0x7fd5980cff00 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0097a0 tx=0x7fd58c02a1a0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.453+0000 7fd59e5a2640 1 -- 172.21.15.77:0/1410206137 shutdown_connections 2024-04-25T01:24:06.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.453+0000 7fd59e5a2640 1 --2- 172.21.15.77:0/1410206137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5980b2ad0 0x7fd5980cff00 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.453+0000 7fd59e5a2640 1 -- 172.21.15.77:0/1410206137 >> 172.21.15.77:0/1410206137 conn(0x7fd59806b540 msgr2=0x7fd59806b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.453+0000 7fd59e5a2640 1 -- 172.21.15.77:0/1410206137 shutdown_connections 2024-04-25T01:24:06.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.454+0000 7fd59e5a2640 1 -- 172.21.15.77:0/1410206137 wait complete. 2024-04-25T01:24:06.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.455+0000 7fd59e5a2640 1 Processor -- start 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.455+0000 7fd59e5a2640 1 -- start start 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.455+0000 7fd59e5a2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd5980b2ad0 0x7fd5981465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.455+0000 7fd59e5a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd598158700 0x7fd5980af640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd59e5a2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd598161b80 0x7fd5980ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd59e5a2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5980d0440 con 0x7fd598158700 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd59e5a2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd59806cce0 con 0x7fd598161b80 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd59e5a2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd5980d05c0 con 0x7fd5980b2ad0 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd5977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd598158700 0x7fd5980af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd5977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd598158700 0x7fd5980af640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45596/0 (socket says 172.21.15.77:45596) 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd5977fe640 1 -- 172.21.15.77:0/2095227299 learned_addr learned my addr 172.21.15.77:0/2095227299 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd597fff640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd5980b2ad0 0x7fd5981465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd59cb3f640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd598161b80 0x7fd5980ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd5977fe640 1 -- 172.21.15.77:0/2095227299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd5980b2ad0 msgr2=0x7fd5981465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd5977fe640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd5980b2ad0 0x7fd5981465c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd5977fe640 1 -- 172.21.15.77:0/2095227299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd598161b80 msgr2=0x7fd5980ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd5977fe640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd598161b80 0x7fd5980ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd5977fe640 1 -- 172.21.15.77:0/2095227299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5981f83e0 con 0x7fd598158700 2024-04-25T01:24:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd597fff640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd5980b2ad0 0x7fd5981465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd5977fe640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd598158700 0x7fd5980af640 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd58c000c00 tx=0x7fd58c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd594ff9640 1 -- 172.21.15.77:0/2095227299 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58c037040 con 0x7fd598158700 2024-04-25T01:24:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd594ff9640 1 -- 172.21.15.77:0/2095227299 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd58c042070 con 0x7fd598158700 2024-04-25T01:24:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd59e5a2640 1 -- 172.21.15.77:0/2095227299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5980afb80 con 0x7fd598158700 2024-04-25T01:24:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd594ff9640 1 -- 172.21.15.77:0/2095227299 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58c030a70 con 0x7fd598158700 2024-04-25T01:24:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.456+0000 7fd59e5a2640 1 -- 172.21.15.77:0/2095227299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5981f7b40 con 0x7fd598158700 2024-04-25T01:24:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.457+0000 7fd59e5a2640 1 -- 172.21.15.77:0/2095227299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5980679d0 con 0x7fd598158700 2024-04-25T01:24:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.458+0000 7fd594ff9640 1 -- 172.21.15.77:0/2095227299 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd58c030c10 con 0x7fd598158700 2024-04-25T01:24:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.458+0000 7fd594ff9640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd58406be30 0x7fd58406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.458+0000 7fd594ff9640 1 -- 172.21.15.77:0/2095227299 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7fd58c0abb70 con 0x7fd598158700 2024-04-25T01:24:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.458+0000 7fd597fff640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd58406be30 0x7fd58406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.459+0000 7fd597fff640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd58406be30 0x7fd58406e2f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd588000f60 tx=0x7fd588009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.463+0000 7fd594ff9640 1 -- 172.21.15.77:0/2095227299 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd5980679d0 con 0x7fd598158700 2024-04-25T01:24:06.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.669+0000 7fd59e5a2640 1 -- 172.21.15.77:0/2095227299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd59806b920 con 0x7fd598158700 2024-04-25T01:24:06.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.669+0000 7fd594ff9640 1 -- 172.21.15.77:0/2095227299 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd59806b920 con 0x7fd598158700 2024-04-25T01:24:06.649 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:06.649 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T01:24:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 -- 172.21.15.77:0/2095227299 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd58406be30 msgr2=0x7fd58406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd58406be30 0x7fd58406e2f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd588000f60 tx=0x7fd588009040 comp rx=0 tx=0).stop 2024-04-25T01:24:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 -- 172.21.15.77:0/2095227299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd598158700 msgr2=0x7fd5980af640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd598158700 0x7fd5980af640 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd58c000c00 tx=0x7fd58c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 -- 172.21.15.77:0/2095227299 shutdown_connections 2024-04-25T01:24:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd58406be30 0x7fd58406e2f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd598158700 0x7fd5980af640 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd598161b80 0x7fd5980ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 --2- 172.21.15.77:0/2095227299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd5980b2ad0 0x7fd5981465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 -- 172.21.15.77:0/2095227299 >> 172.21.15.77:0/2095227299 conn(0x7fd59806b540 msgr2=0x7fd598160f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 -- 172.21.15.77:0/2095227299 shutdown_connections 2024-04-25T01:24:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.671+0000 7fd59e5a2640 1 -- 172.21.15.77:0/2095227299 wait complete. 2024-04-25T01:24:06.662 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-25T01:24:06.662 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T01:24:06.662 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:24:06.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:24:06.747 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T01:24:06.747 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:24:06.747 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport os\nimport sys\nimport time\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/background_file", os.O_RDWR | os.O_CREAT, 0o644)\ntry:\n while True:\n print("write_background: writing", file=sys.stderr)\n os.write(fd, b\'content\')\n if not True:\n break\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\nexcept IOError as e:\n pass\nos.close(fd)\n'], check_status=True, hostname='smithi077') 2024-04-25T01:24:06.747 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:24:06.747 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:24:06.762 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:24:06.763 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:24:06.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:24:06.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.975+0000 7f948df7e640 1 Processor -- start 2024-04-25T01:24:06.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.975+0000 7f948df7e640 1 -- start start 2024-04-25T01:24:06.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f948df7e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9488145590 0x7f94880b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f948df7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f948815f9b0 0x7f948815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f948df7e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f94880ce110 0x7f94880ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f948df7e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f9488160830 con 0x7f9488145590 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f948df7e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9488160350 con 0x7f94880ce110 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f948df7e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f948806cde0 con 0x7f948815f9b0 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f94877fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9488145590 0x7f94880b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f94877fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9488145590 0x7f94880b4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48384/0 (socket says 172.21.15.77:48384) 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f94877fe640 1 -- 172.21.15.77:0/3228903982 learned_addr learned my addr 172.21.15.77:0/3228903982 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f9486ffd640 1 --2- 172.21.15.77:0/3228903982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f948815f9b0 0x7f948815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f9487fff640 1 --2- 172.21.15.77:0/3228903982 >> v2:172.21.15.121:3300/0 conn(0x7f94880ce110 0x7f94880ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f94877fe640 1 -- 172.21.15.77:0/3228903982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f948815f9b0 msgr2=0x7f948815fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f94877fe640 1 --2- 172.21.15.77:0/3228903982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f948815f9b0 0x7f948815fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f94877fe640 1 -- 172.21.15.77:0/3228903982 >> v2:172.21.15.121:3300/0 conn(0x7f94880ce110 msgr2=0x7f94880ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:06.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f94877fe640 1 --2- 172.21.15.77:0/3228903982 >> v2:172.21.15.121:3300/0 conn(0x7f94880ce110 0x7f94880ce5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.976+0000 7f94877fe640 1 -- 172.21.15.77:0/3228903982 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94880b4240 con 0x7f9488145590 2024-04-25T01:24:06.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f94877fe640 1 --2- 172.21.15.77:0/3228903982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9488145590 0x7f94880b4a20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f948806ea20 tx=0x7f947802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8498b3c2784de7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f94867fc640 1 -- 172.21.15.77:0/3228903982 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478037070 con 0x7f9488145590 2024-04-25T01:24:06.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f94867fc640 1 -- 172.21.15.77:0/3228903982 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9478032440 con 0x7f9488145590 2024-04-25T01:24:06.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f94867fc640 1 -- 172.21.15.77:0/3228903982 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478032810 con 0x7f9488145590 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f948df7e640 1 -- 172.21.15.77:0/3228903982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9488145590 msgr2=0x7f94880b4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f948df7e640 1 --2- 172.21.15.77:0/3228903982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9488145590 0x7f94880b4a20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f948806ea20 tx=0x7f947802a1e0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f948df7e640 1 -- 172.21.15.77:0/3228903982 shutdown_connections 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f948df7e640 1 --2- 172.21.15.77:0/3228903982 >> v2:172.21.15.121:3300/0 conn(0x7f94880ce110 0x7f94880ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f948df7e640 1 --2- 172.21.15.77:0/3228903982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f948815f9b0 0x7f948815fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f948df7e640 1 --2- 172.21.15.77:0/3228903982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9488145590 0x7f94880b4a20 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f948df7e640 1 -- 172.21.15.77:0/3228903982 >> 172.21.15.77:0/3228903982 conn(0x7f94880b1820 msgr2=0x7f94880b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f948df7e640 1 -- 172.21.15.77:0/3228903982 shutdown_connections 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.977+0000 7f948df7e640 1 -- 172.21.15.77:0/3228903982 wait complete. 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.978+0000 7f948df7e640 1 Processor -- start 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.979+0000 7f948df7e640 1 -- start start 2024-04-25T01:24:06.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.979+0000 7f948df7e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94880ce110 0x7f94881f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.979+0000 7f948df7e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9488145590 0x7f948806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.979+0000 7f948df7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f948815f9b0 0x7f94880a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.979+0000 7f948df7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f94880b3c80 con 0x7f948815f9b0 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.979+0000 7f948df7e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f94880b39a0 con 0x7f94880ce110 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.979+0000 7f948df7e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f94880b3f60 con 0x7f9488145590 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.979+0000 7f9487fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f948815f9b0 0x7f94880a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f9487fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f948815f9b0 0x7f94880a9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48392/0 (socket says 172.21.15.77:48392) 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f9487fff640 1 -- 172.21.15.77:0/1071278580 learned_addr learned my addr 172.21.15.77:0/1071278580 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f94877fe640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94880ce110 0x7f94881f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f9486ffd640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9488145590 0x7f948806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f9487fff640 1 -- 172.21.15.77:0/1071278580 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9488145590 msgr2=0x7f948806d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f9487fff640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9488145590 0x7f948806d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f9487fff640 1 -- 172.21.15.77:0/1071278580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94880ce110 msgr2=0x7f94881f4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f9487fff640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94880ce110 0x7f94881f4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f9487fff640 1 -- 172.21.15.77:0/1071278580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94881f3c10 con 0x7f948815f9b0 2024-04-25T01:24:06.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f9486ffd640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9488145590 0x7f948806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:06.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f9487fff640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f948815f9b0 0x7f94880a9d50 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9480007e60 tx=0x7f948000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f946ffff640 1 -- 172.21.15.77:0/1071278580 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480014620 con 0x7f948815f9b0 2024-04-25T01:24:06.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f946ffff640 1 -- 172.21.15.77:0/1071278580 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9480019070 con 0x7f948815f9b0 2024-04-25T01:24:06.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f946ffff640 1 -- 172.21.15.77:0/1071278580 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480014a20 con 0x7f948815f9b0 2024-04-25T01:24:06.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f948df7e640 1 -- 172.21.15.77:0/1071278580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94881f41d0 con 0x7f948815f9b0 2024-04-25T01:24:06.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.980+0000 7f948df7e640 1 -- 172.21.15.77:0/1071278580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94881f4790 con 0x7f948815f9b0 2024-04-25T01:24:06.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.981+0000 7f948df7e640 1 -- 172.21.15.77:0/1071278580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94881f4990 con 0x7f948815f9b0 2024-04-25T01:24:06.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.982+0000 7f946ffff640 1 -- 172.21.15.77:0/1071278580 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9480014bc0 con 0x7f948815f9b0 2024-04-25T01:24:06.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.982+0000 7f946ffff640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f947406be30 0x7f947406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.982+0000 7f946ffff640 1 -- 172.21.15.77:0/1071278580 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f948008b970 con 0x7f948815f9b0 2024-04-25T01:24:06.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.982+0000 7f94877fe640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f947406be30 0x7f947406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.983+0000 7f94877fe640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f947406be30 0x7f947406e2f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9478034000 tx=0x7f947802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:06.986+0000 7f946ffff640 1 -- 172.21.15.77:0/1071278580 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f94881f4990 con 0x7f948815f9b0 2024-04-25T01:24:07.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.162+0000 7f948df7e640 1 -- 172.21.15.77:0/1071278580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f948813f8c0 con 0x7f948815f9b0 2024-04-25T01:24:07.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.163+0000 7f946ffff640 1 -- 172.21.15.77:0/1071278580 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+17203 (secure 0 0 0) 0x7f948005a0e0 con 0x7f948815f9b0 2024-04-25T01:24:07.144 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:07.144 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":55,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:24:01.947173+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:23:26.392274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T01:23:27.146093+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6836/3189776231":"2024-04-26T01:24:01.947151+0000","172.21.15.77:6834/3189776231":"2024-04-26T01:24:01.947151+0000","172.21.15.121:6834/3834135103":"2024-04-26T01:24:00.948815+0000","172.21.15.121:6832/3834135103":"2024-04-26T01:24:00.948815+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:24:07.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.165+0000 7f948df7e640 1 -- 172.21.15.77:0/1071278580 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f947406be30 msgr2=0x7f947406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.165+0000 7f948df7e640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f947406be30 0x7f947406e2f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9478034000 tx=0x7f947802e000 comp rx=0 tx=0).stop 2024-04-25T01:24:07.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.165+0000 7f948df7e640 1 -- 172.21.15.77:0/1071278580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f948815f9b0 msgr2=0x7f94880a9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.165+0000 7f948df7e640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f948815f9b0 0x7f94880a9d50 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9480007e60 tx=0x7f948000e4b0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.165+0000 7f948df7e640 1 -- 172.21.15.77:0/1071278580 shutdown_connections 2024-04-25T01:24:07.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.165+0000 7f948df7e640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f947406be30 0x7f947406e2f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.165+0000 7f948df7e640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f948815f9b0 0x7f94880a9d50 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.165+0000 7f948df7e640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9488145590 0x7f948806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.165+0000 7f948df7e640 1 --2- 172.21.15.77:0/1071278580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94880ce110 0x7f94881f4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.166+0000 7f948df7e640 1 -- 172.21.15.77:0/1071278580 >> 172.21.15.77:0/1071278580 conn(0x7f94880b1820 msgr2=0x7f9488141c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:07.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.166+0000 7f948df7e640 1 -- 172.21.15.77:0/1071278580 shutdown_connections 2024-04-25T01:24:07.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.166+0000 7f948df7e640 1 -- 172.21.15.77:0/1071278580 wait complete. 2024-04-25T01:24:07.156 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:23:26.392274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:23:27.146093+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:24:07.157 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:24:07.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.276+0000 7f45d0858640 1 Processor -- start 2024-04-25T01:24:07.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.276+0000 7f45d0858640 1 -- start start 2024-04-25T01:24:07.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45d0858640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f45cc15fc40 0x7f45cc1600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45d0858640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45cc0cedc0 0x7f45cc0cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45d0858640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc06d110 con 0x7f45cc0722e0 2024-04-25T01:24:07.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45d0858640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f45cc160aa0 con 0x7f45cc0cedc0 2024-04-25T01:24:07.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45d0858640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f45cc06da40 con 0x7f45cc15fc40 2024-04-25T01:24:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45cad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45cc0cedc0 0x7f45cc0cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45cad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45cc0cedc0 0x7f45cc0cf1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48406/0 (socket says 172.21.15.77:48406) 2024-04-25T01:24:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45cad9d640 1 -- 172.21.15.77:0/1693422119 learned_addr learned my addr 172.21.15.77:0/1693422119 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45c9d9b640 1 --2- 172.21.15.77:0/1693422119 >> v2:172.21.15.121:3300/0 conn(0x7f45cc15fc40 0x7f45cc1600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45cad9d640 1 -- 172.21.15.77:0/1693422119 >> v2:172.21.15.121:3300/0 conn(0x7f45cc15fc40 msgr2=0x7f45cc1600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45cad9d640 1 --2- 172.21.15.77:0/1693422119 >> v2:172.21.15.121:3300/0 conn(0x7f45cc15fc40 0x7f45cc1600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.277+0000 7f45cad9d640 1 -- 172.21.15.77:0/1693422119 >> v1:172.21.15.121:6789/0 conn(0x7f45cc0722e0 legacy=0x7f45cc06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.278+0000 7f45cad9d640 1 -- 172.21.15.77:0/1693422119 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc06d3f0 con 0x7f45cc0cedc0 2024-04-25T01:24:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.278+0000 7f45cad9d640 1 --2- 172.21.15.77:0/1693422119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45cc0cedc0 0x7f45cc0cf1f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0ce0a0 tx=0x7f45c402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ef8cb2004028d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.278+0000 7f45c959a640 1 -- 172.21.15.77:0/1693422119 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4037070 con 0x7f45cc0cedc0 2024-04-25T01:24:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.278+0000 7f45c959a640 1 -- 172.21.15.77:0/1693422119 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c4032440 con 0x7f45cc0cedc0 2024-04-25T01:24:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.278+0000 7f45c959a640 1 -- 172.21.15.77:0/1693422119 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4032810 con 0x7f45cc0cedc0 2024-04-25T01:24:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.279+0000 7f45d0858640 1 -- 172.21.15.77:0/1693422119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45cc0cedc0 msgr2=0x7f45cc0cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.279+0000 7f45d0858640 1 --2- 172.21.15.77:0/1693422119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45cc0cedc0 0x7f45cc0cf1f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0ce0a0 tx=0x7f45c402a1a0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.279+0000 7f45d0858640 1 -- 172.21.15.77:0/1693422119 shutdown_connections 2024-04-25T01:24:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.279+0000 7f45d0858640 1 --2- 172.21.15.77:0/1693422119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45cc0cedc0 0x7f45cc0cf1f0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.279+0000 7f45d0858640 1 --2- 172.21.15.77:0/1693422119 >> v2:172.21.15.121:3300/0 conn(0x7f45cc15fc40 0x7f45cc1600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.279+0000 7f45d0858640 1 -- 172.21.15.77:0/1693422119 >> 172.21.15.77:0/1693422119 conn(0x7f45cc06b720 msgr2=0x7f45cc0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.279+0000 7f45d0858640 1 -- 172.21.15.77:0/1693422119 shutdown_connections 2024-04-25T01:24:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.279+0000 7f45d0858640 1 -- 172.21.15.77:0/1693422119 wait complete. 2024-04-25T01:24:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.280+0000 7f45d0858640 1 Processor -- start 2024-04-25T01:24:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.280+0000 7f45d0858640 1 -- start start 2024-04-25T01:24:07.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45d0858640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45cc0722e0 0x7f45cc1e6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45d0858640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45cc0cedc0 0x7f45cc156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45d0858640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f45cc15fc40 0x7f45cc1e72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45d0858640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc06cec0 con 0x7f45cc0cedc0 2024-04-25T01:24:07.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45d0858640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc0cf730 con 0x7f45cc0722e0 2024-04-25T01:24:07.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45d0858640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f45cc070b50 con 0x7f45cc15fc40 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45c9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45cc0cedc0 0x7f45cc156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45c9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45cc0cedc0 0x7f45cc156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48408/0 (socket says 172.21.15.77:48408) 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45c9d9b640 1 -- 172.21.15.77:0/1229165813 learned_addr learned my addr 172.21.15.77:0/1229165813 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45cad9d640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f45cc15fc40 0x7f45cc1e72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45ca59c640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45cc0722e0 0x7f45cc1e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45c9d9b640 1 -- 172.21.15.77:0/1229165813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f45cc15fc40 msgr2=0x7f45cc1e72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45c9d9b640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f45cc15fc40 0x7f45cc1e72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45c9d9b640 1 -- 172.21.15.77:0/1229165813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45cc0722e0 msgr2=0x7f45cc1e6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45c9d9b640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45cc0722e0 0x7f45cc1e6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45c9d9b640 1 -- 172.21.15.77:0/1229165813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc157250 con 0x7f45cc0cedc0 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.281+0000 7f45cad9d640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f45cc15fc40 0x7f45cc1e72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.282+0000 7f45c9d9b640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45cc0cedc0 0x7f45cc156d10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f45c000d5f0 tx=0x7f45c000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.282+0000 7f45baffd640 1 -- 172.21.15.77:0/1229165813 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c00125f0 con 0x7f45cc0cedc0 2024-04-25T01:24:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.282+0000 7f45baffd640 1 -- 172.21.15.77:0/1229165813 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c0019070 con 0x7f45cc0cedc0 2024-04-25T01:24:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.282+0000 7f45baffd640 1 -- 172.21.15.77:0/1229165813 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c00129d0 con 0x7f45cc0cedc0 2024-04-25T01:24:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.282+0000 7f45d0858640 1 -- 172.21.15.77:0/1229165813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45cc1574f0 con 0x7f45cc0cedc0 2024-04-25T01:24:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.282+0000 7f45d0858640 1 -- 172.21.15.77:0/1229165813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45cc157d90 con 0x7f45cc0cedc0 2024-04-25T01:24:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.283+0000 7f45d0858640 1 -- 172.21.15.77:0/1229165813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45cc068fb0 con 0x7f45cc0cedc0 2024-04-25T01:24:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.283+0000 7f45baffd640 1 -- 172.21.15.77:0/1229165813 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f45c0012b70 con 0x7f45cc0cedc0 2024-04-25T01:24:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.283+0000 7f45baffd640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f45ac06be30 0x7f45ac06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.284+0000 7f45baffd640 1 -- 172.21.15.77:0/1229165813 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f45c008b420 con 0x7f45cc0cedc0 2024-04-25T01:24:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.284+0000 7f45ca59c640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f45ac06be30 0x7f45ac06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.284+0000 7f45ca59c640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f45ac06be30 0x7f45ac06e2f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f45bc005520 tx=0x7f45bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:07.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.288+0000 7f45baffd640 1 -- 172.21.15.77:0/1229165813 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f45c0059c20 con 0x7f45cc0cedc0 2024-04-25T01:24:07.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.462+0000 7f45d0858640 1 -- 172.21.15.77:0/1229165813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f45cc072c90 con 0x7f45cc0cedc0 2024-04-25T01:24:07.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.463+0000 7f45baffd640 1 -- 172.21.15.77:0/1229165813 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) ==== 93+0+10 (secure 0 0 0) 0x7f45c0059520 con 0x7f45cc0cedc0 2024-04-25T01:24:07.443 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 -- 172.21.15.77:0/1229165813 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f45ac06be30 msgr2=0x7f45ac06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f45ac06be30 0x7f45ac06e2f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f45bc005520 tx=0x7f45bc009040 comp rx=0 tx=0).stop 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 -- 172.21.15.77:0/1229165813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45cc0cedc0 msgr2=0x7f45cc156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45cc0cedc0 0x7f45cc156d10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f45c000d5f0 tx=0x7f45c000dab0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 -- 172.21.15.77:0/1229165813 shutdown_connections 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f45ac06be30 0x7f45ac06e2f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45cc0cedc0 0x7f45cc156d10 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f45cc15fc40 0x7f45cc1e72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 --2- 172.21.15.77:0/1229165813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45cc0722e0 0x7f45cc1e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 -- 172.21.15.77:0/1229165813 >> 172.21.15.77:0/1229165813 conn(0x7f45cc06b720 msgr2=0x7f45cc0ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 -- 172.21.15.77:0/1229165813 shutdown_connections 2024-04-25T01:24:07.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.465+0000 7f45d0858640 1 -- 172.21.15.77:0/1229165813 wait complete. 2024-04-25T01:24:07.455 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:24:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.575+0000 7fef253ab640 1 Processor -- start 2024-04-25T01:24:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.575+0000 7fef253ab640 1 -- start start 2024-04-25T01:24:07.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef253ab640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fef200d0940 0x7fef2006ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef253ab640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef200b0c50 0x7fef2006f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef253ab640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef200d2810 con 0x7fef2015c2b0 2024-04-25T01:24:07.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef253ab640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fef200cd920 con 0x7fef200d0940 2024-04-25T01:24:07.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef253ab640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fef200cd2d0 con 0x7fef200b0c50 2024-04-25T01:24:07.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef167fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fef200d0940 0x7fef2006ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef167fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fef200d0940 0x7fef2006ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48422/0 (socket says 172.21.15.77:48422) 2024-04-25T01:24:07.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef167fc640 1 -- 172.21.15.77:0/376088496 learned_addr learned my addr 172.21.15.77:0/376088496 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:07.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef167fc640 1 -- 172.21.15.77:0/376088496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef200b0c50 msgr2=0x7fef2006f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:07.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef167fc640 1 --2- 172.21.15.77:0/376088496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef200b0c50 0x7fef2006f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef167fc640 1 -- 172.21.15.77:0/376088496 >> v1:172.21.15.121:6789/0 conn(0x7fef2015c2b0 legacy=0x7fef200d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:07.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.576+0000 7fef167fc640 1 -- 172.21.15.77:0/376088496 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef200cd5b0 con 0x7fef200d0940 2024-04-25T01:24:07.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.577+0000 7fef167fc640 1 --2- 172.21.15.77:0/376088496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef200d0940 0x7fef2006ef40 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fef08009870 tx=0x7fef0802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2057c8f7453720df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:07.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.577+0000 7fef1e7fc640 1 -- 172.21.15.77:0/376088496 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08037070 con 0x7fef200d0940 2024-04-25T01:24:07.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.577+0000 7fef1e7fc640 1 -- 172.21.15.77:0/376088496 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef08032440 con 0x7fef200d0940 2024-04-25T01:24:07.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.577+0000 7fef1e7fc640 1 -- 172.21.15.77:0/376088496 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08032810 con 0x7fef200d0940 2024-04-25T01:24:07.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.578+0000 7fef253ab640 1 -- 172.21.15.77:0/376088496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef200d0940 msgr2=0x7fef2006ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.578+0000 7fef253ab640 1 --2- 172.21.15.77:0/376088496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef200d0940 0x7fef2006ef40 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fef08009870 tx=0x7fef0802a210 comp rx=0 tx=0).stop 2024-04-25T01:24:07.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.578+0000 7fef253ab640 1 -- 172.21.15.77:0/376088496 shutdown_connections 2024-04-25T01:24:07.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.578+0000 7fef253ab640 1 --2- 172.21.15.77:0/376088496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef200b0c50 0x7fef2006f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.578+0000 7fef253ab640 1 --2- 172.21.15.77:0/376088496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef200d0940 0x7fef2006ef40 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.578+0000 7fef253ab640 1 -- 172.21.15.77:0/376088496 >> 172.21.15.77:0/376088496 conn(0x7fef200d1c50 msgr2=0x7fef200d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:07.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.578+0000 7fef253ab640 1 -- 172.21.15.77:0/376088496 shutdown_connections 2024-04-25T01:24:07.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.578+0000 7fef253ab640 1 -- 172.21.15.77:0/376088496 wait complete. 2024-04-25T01:24:07.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.579+0000 7fef253ab640 1 Processor -- start 2024-04-25T01:24:07.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.579+0000 7fef253ab640 1 -- start start 2024-04-25T01:24:07.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef253ab640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef200b0c50 0x7fef200ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef253ab640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef200d0940 0x7fef201f4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef253ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef2015c2b0 0x7fef20150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef253ab640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fef200b14d0 con 0x7fef2015c2b0 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef253ab640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fef200b1350 con 0x7fef200b0c50 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef253ab640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fef200cfa30 con 0x7fef200d0940 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef1f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef2015c2b0 0x7fef20150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef1f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef2015c2b0 0x7fef20150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48430/0 (socket says 172.21.15.77:48430) 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef1f7fe640 1 -- 172.21.15.77:0/796837288 learned_addr learned my addr 172.21.15.77:0/796837288 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef1effd640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef200b0c50 0x7fef200ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef167fc640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef200d0940 0x7fef201f4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef1f7fe640 1 -- 172.21.15.77:0/796837288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef200d0940 msgr2=0x7fef201f4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef1f7fe640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef200d0940 0x7fef201f4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef1f7fe640 1 -- 172.21.15.77:0/796837288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef200b0c50 msgr2=0x7fef200ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef1f7fe640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef200b0c50 0x7fef200ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef1f7fe640 1 -- 172.21.15.77:0/796837288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef201f5490 con 0x7fef2015c2b0 2024-04-25T01:24:07.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef167fc640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef200d0940 0x7fef201f4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:07.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.580+0000 7fef1f7fe640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef2015c2b0 0x7fef20150270 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fef1800b970 tx=0x7fef1800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:07.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.581+0000 7fef17fff640 1 -- 172.21.15.77:0/796837288 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef18019070 con 0x7fef2015c2b0 2024-04-25T01:24:07.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.581+0000 7fef17fff640 1 -- 172.21.15.77:0/796837288 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef1800cb60 con 0x7fef2015c2b0 2024-04-25T01:24:07.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.581+0000 7fef253ab640 1 -- 172.21.15.77:0/796837288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef2015c690 con 0x7fef2015c2b0 2024-04-25T01:24:07.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.581+0000 7fef17fff640 1 -- 172.21.15.77:0/796837288 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef18014440 con 0x7fef2015c2b0 2024-04-25T01:24:07.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.581+0000 7fef253ab640 1 -- 172.21.15.77:0/796837288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef201f5cf0 con 0x7fef2015c2b0 2024-04-25T01:24:07.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.582+0000 7fef253ab640 1 -- 172.21.15.77:0/796837288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef201f5790 con 0x7fef2015c2b0 2024-04-25T01:24:07.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.582+0000 7fef17fff640 1 -- 172.21.15.77:0/796837288 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fef1801c070 con 0x7fef2015c2b0 2024-04-25T01:24:07.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.582+0000 7fef17fff640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fef0006be30 0x7fef0006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.582+0000 7fef17fff640 1 -- 172.21.15.77:0/796837288 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7fef1808acc0 con 0x7fef2015c2b0 2024-04-25T01:24:07.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.582+0000 7fef1effd640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fef0006be30 0x7fef0006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.583+0000 7fef1effd640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fef0006be30 0x7fef0006e2f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fef10006fd0 tx=0x7fef10009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:07.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.587+0000 7fef17fff640 1 -- 172.21.15.77:0/796837288 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fef201f5790 con 0x7fef2015c2b0 2024-04-25T01:24:07.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.761+0000 7fef253ab640 1 -- 172.21.15.77:0/796837288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fef201572d0 con 0x7fef2015c2b0 2024-04-25T01:24:07.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.762+0000 7fef17fff640 1 -- 172.21.15.77:0/796837288 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) ==== 104+0+11 (secure 0 0 0) 0x7fef18058e70 con 0x7fef2015c2b0 2024-04-25T01:24:07.742 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:24:07.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 -- 172.21.15.77:0/796837288 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fef0006be30 msgr2=0x7fef0006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fef0006be30 0x7fef0006e2f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fef10006fd0 tx=0x7fef10009060 comp rx=0 tx=0).stop 2024-04-25T01:24:07.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 -- 172.21.15.77:0/796837288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef2015c2b0 msgr2=0x7fef20150270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef2015c2b0 0x7fef20150270 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fef1800b970 tx=0x7fef1800be30 comp rx=0 tx=0).stop 2024-04-25T01:24:07.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 -- 172.21.15.77:0/796837288 shutdown_connections 2024-04-25T01:24:07.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fef0006be30 0x7fef0006e2f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef2015c2b0 0x7fef20150270 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef200d0940 0x7fef201f4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 --2- 172.21.15.77:0/796837288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef200b0c50 0x7fef200ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 -- 172.21.15.77:0/796837288 >> 172.21.15.77:0/796837288 conn(0x7fef200d1c50 msgr2=0x7fef2006ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:07.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 -- 172.21.15.77:0/796837288 shutdown_connections 2024-04-25T01:24:07.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.764+0000 7fef253ab640 1 -- 172.21.15.77:0/796837288 wait complete. 2024-04-25T01:24:07.755 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:24:07.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.875+0000 7f34eb02a640 1 Processor -- start 2024-04-25T01:24:07.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.875+0000 7f34eb02a640 1 -- start start 2024-04-25T01:24:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.876+0000 7f34eb02a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34e4158a70 0x7f34e40b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.876+0000 7f34eb02a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34e415c210 con 0x7f34e40b37c0 2024-04-25T01:24:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.876+0000 7f34eb02a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34e40b2ce0 con 0x7f34e415bc30 2024-04-25T01:24:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.876+0000 7f34eb02a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f34e40b2fc0 con 0x7f34e4158a70 2024-04-25T01:24:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.876+0000 7f34e8dc6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34e4158a70 0x7f34e40b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.876+0000 7f34e3fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f34e40b37c0 0x7f34e415b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49142/0 (socket says 172.21.15.77:49142) 2024-04-25T01:24:07.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.876+0000 7f34e3fff640 1 -- 172.21.15.77:0/3020378625 learned_addr learned my addr 172.21.15.77:0/3020378625 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:07.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.876+0000 7f34e8dc6640 1 -- 172.21.15.77:0/3020378625 >> v1:172.21.15.121:6789/0 conn(0x7f34e415bc30 legacy=0x7f34e406c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:07.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.876+0000 7f34e8dc6640 1 -- 172.21.15.77:0/3020378625 >> v1:172.21.15.77:6789/0 conn(0x7f34e40b37c0 legacy=0x7f34e415b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.876+0000 7f34e8dc6640 1 -- 172.21.15.77:0/3020378625 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e40b32a0 con 0x7f34e4158a70 2024-04-25T01:24:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.877+0000 7f34e8dc6640 1 --2- 172.21.15.77:0/3020378625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e4158a70 0x7f34e40b40c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f34d40097d0 tx=0x7f34d402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac5a539a0b036f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.877+0000 7f34e37fe640 1 -- 172.21.15.77:0/3020378625 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4037070 con 0x7f34e4158a70 2024-04-25T01:24:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.877+0000 7f34e37fe640 1 -- 172.21.15.77:0/3020378625 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d4032440 con 0x7f34e4158a70 2024-04-25T01:24:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.877+0000 7f34e37fe640 1 -- 172.21.15.77:0/3020378625 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4032810 con 0x7f34e4158a70 2024-04-25T01:24:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.878+0000 7f34eb02a640 1 -- 172.21.15.77:0/3020378625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e4158a70 msgr2=0x7f34e40b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.878+0000 7f34eb02a640 1 --2- 172.21.15.77:0/3020378625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e4158a70 0x7f34e40b40c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f34d40097d0 tx=0x7f34d402a210 comp rx=0 tx=0).stop 2024-04-25T01:24:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.878+0000 7f34eb02a640 1 -- 172.21.15.77:0/3020378625 shutdown_connections 2024-04-25T01:24:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.878+0000 7f34eb02a640 1 --2- 172.21.15.77:0/3020378625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e4158a70 0x7f34e40b40c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.878+0000 7f34eb02a640 1 -- 172.21.15.77:0/3020378625 >> 172.21.15.77:0/3020378625 conn(0x7f34e406b720 msgr2=0x7f34e40b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:07.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.878+0000 7f34eb02a640 1 -- 172.21.15.77:0/3020378625 shutdown_connections 2024-04-25T01:24:07.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.878+0000 7f34eb02a640 1 -- 172.21.15.77:0/3020378625 wait complete. 2024-04-25T01:24:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.879+0000 7f34eb02a640 1 Processor -- start 2024-04-25T01:24:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.879+0000 7f34eb02a640 1 -- start start 2024-04-25T01:24:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34eb02a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e40b37c0 0x7f34e4157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34eb02a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34e4158a70 0x7f34e4135600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34eb02a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34e415bc30 0x7f34e40a9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34eb02a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f34e40b4780 con 0x7f34e40b37c0 2024-04-25T01:24:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34eb02a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f34e40b4600 con 0x7f34e415bc30 2024-04-25T01:24:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34eb02a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f34e40b4900 con 0x7f34e4158a70 2024-04-25T01:24:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e8dc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e40b37c0 0x7f34e4157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e8dc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e40b37c0 0x7f34e4157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48442/0 (socket says 172.21.15.77:48442) 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e8dc6640 1 -- 172.21.15.77:0/3351114973 learned_addr learned my addr 172.21.15.77:0/3351114973 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e95c7640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34e415bc30 0x7f34e40a9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e3fff640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34e4158a70 0x7f34e4135600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e8dc6640 1 -- 172.21.15.77:0/3351114973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34e4158a70 msgr2=0x7f34e4135600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e8dc6640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34e4158a70 0x7f34e4135600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e8dc6640 1 -- 172.21.15.77:0/3351114973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34e415bc30 msgr2=0x7f34e40a9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e8dc6640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34e415bc30 0x7f34e40a9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e8dc6640 1 -- 172.21.15.77:0/3351114973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e41322b0 con 0x7f34e40b37c0 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e3fff640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34e4158a70 0x7f34e4135600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e95c7640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34e415bc30 0x7f34e40a9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.880+0000 7f34e8dc6640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e40b37c0 0x7f34e4157830 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f34d4000c00 tx=0x7f34d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.881+0000 7f34e17fa640 1 -- 172.21.15.77:0/3351114973 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4037040 con 0x7f34e40b37c0 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.881+0000 7f34e17fa640 1 -- 172.21.15.77:0/3351114973 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d4004230 con 0x7f34e40b37c0 2024-04-25T01:24:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.881+0000 7f34eb02a640 1 -- 172.21.15.77:0/3351114973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e4132590 con 0x7f34e40b37c0 2024-04-25T01:24:07.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.881+0000 7f34e17fa640 1 -- 172.21.15.77:0/3351114973 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4032e20 con 0x7f34e40b37c0 2024-04-25T01:24:07.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.881+0000 7f34eb02a640 1 -- 172.21.15.77:0/3351114973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e4132b50 con 0x7f34e40b37c0 2024-04-25T01:24:07.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.882+0000 7f34eb02a640 1 -- 172.21.15.77:0/3351114973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e4068fb0 con 0x7f34e40b37c0 2024-04-25T01:24:07.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.882+0000 7f34e17fa640 1 -- 172.21.15.77:0/3351114973 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f34d4030770 con 0x7f34e40b37c0 2024-04-25T01:24:07.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.882+0000 7f34e17fa640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34d006be30 0x7f34d006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:07.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.882+0000 7f34e17fa640 1 -- 172.21.15.77:0/3351114973 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f34d40abfb0 con 0x7f34e40b37c0 2024-04-25T01:24:07.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.883+0000 7f34e3fff640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34d006be30 0x7f34d006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:07.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.883+0000 7f34e3fff640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34d006be30 0x7f34d006e2f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f34e40af620 tx=0x7f34d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:07.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:07.887+0000 7f34e17fa640 1 -- 172.21.15.77:0/3351114973 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f34d407a720 con 0x7f34e40b37c0 2024-04-25T01:24:08.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.061+0000 7f34eb02a640 1 -- 172.21.15.77:0/3351114973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f34e40b3c70 con 0x7f34e40b37c0 2024-04-25T01:24:08.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.062+0000 7f34e17fa640 1 -- 172.21.15.77:0/3351114973 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) ==== 100+0+11 (secure 0 0 0) 0x7f34d407a020 con 0x7f34e40b37c0 2024-04-25T01:24:08.041 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.063+0000 7f34eb02a640 1 -- 172.21.15.77:0/3351114973 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34d006be30 msgr2=0x7f34d006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.063+0000 7f34eb02a640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34d006be30 0x7f34d006e2f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f34e40af620 tx=0x7f34d8009040 comp rx=0 tx=0).stop 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.063+0000 7f34eb02a640 1 -- 172.21.15.77:0/3351114973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e40b37c0 msgr2=0x7f34e4157830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.063+0000 7f34eb02a640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e40b37c0 0x7f34e4157830 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f34d4000c00 tx=0x7f34d40046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.064+0000 7f34eb02a640 1 -- 172.21.15.77:0/3351114973 shutdown_connections 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.064+0000 7f34eb02a640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f34d006be30 0x7f34d006e2f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.064+0000 7f34eb02a640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34e415bc30 0x7f34e40a9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.064+0000 7f34eb02a640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34e4158a70 0x7f34e4135600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.064+0000 7f34eb02a640 1 --2- 172.21.15.77:0/3351114973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e40b37c0 0x7f34e4157830 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.064+0000 7f34eb02a640 1 -- 172.21.15.77:0/3351114973 >> 172.21.15.77:0/3351114973 conn(0x7f34e406b720 msgr2=0x7f34e415b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.064+0000 7f34eb02a640 1 -- 172.21.15.77:0/3351114973 shutdown_connections 2024-04-25T01:24:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.064+0000 7f34eb02a640 1 -- 172.21.15.77:0/3351114973 wait complete. 2024-04-25T01:24:08.054 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:24:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.172+0000 7fd22ba72640 1 Processor -- start 2024-04-25T01:24:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.172+0000 7fd22ba72640 1 -- start start 2024-04-25T01:24:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22ba72640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd224143940 0x7fd22406e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22ba72640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd22415f400 0x7fd22406eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22ba72640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd224070e60 con 0x7fd224143d20 2024-04-25T01:24:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22ba72640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fd224071140 con 0x7fd224143940 2024-04-25T01:24:08.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22ba72640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd224071420 con 0x7fd22415f400 2024-04-25T01:24:08.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22980e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd224143940 0x7fd22406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22980e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd224143940 0x7fd22406e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48446/0 (socket says 172.21.15.77:48446) 2024-04-25T01:24:08.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22980e640 1 -- 172.21.15.77:0/2222532904 learned_addr learned my addr 172.21.15.77:0/2222532904 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:08.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22980e640 1 -- 172.21.15.77:0/2222532904 >> v2:172.21.15.121:3300/0 conn(0x7fd22415f400 msgr2=0x7fd22406eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:08.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22980e640 1 --2- 172.21.15.77:0/2222532904 >> v2:172.21.15.121:3300/0 conn(0x7fd22415f400 0x7fd22406eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22980e640 1 -- 172.21.15.77:0/2222532904 >> v1:172.21.15.121:6789/0 conn(0x7fd224143d20 legacy=0x7fd22415ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:08.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.173+0000 7fd22980e640 1 -- 172.21.15.77:0/2222532904 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd224071700 con 0x7fd224143940 2024-04-25T01:24:08.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.174+0000 7fd22980e640 1 --2- 172.21.15.77:0/2222532904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd224143940 0x7fd22406e650 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd220009e50 tx=0x7fd22002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539f4d18b0812ef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:08.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.174+0000 7fd22880c640 1 -- 172.21.15.77:0/2222532904 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd220036070 con 0x7fd224143940 2024-04-25T01:24:08.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.174+0000 7fd22880c640 1 -- 172.21.15.77:0/2222532904 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd220031600 con 0x7fd224143940 2024-04-25T01:24:08.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.174+0000 7fd22880c640 1 -- 172.21.15.77:0/2222532904 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2200319d0 con 0x7fd224143940 2024-04-25T01:24:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.175+0000 7fd22ba72640 1 -- 172.21.15.77:0/2222532904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd224143940 msgr2=0x7fd22406e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.175+0000 7fd22ba72640 1 --2- 172.21.15.77:0/2222532904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd224143940 0x7fd22406e650 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd220009e50 tx=0x7fd22002a710 comp rx=0 tx=0).stop 2024-04-25T01:24:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.175+0000 7fd22ba72640 1 -- 172.21.15.77:0/2222532904 shutdown_connections 2024-04-25T01:24:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.175+0000 7fd22ba72640 1 --2- 172.21.15.77:0/2222532904 >> v2:172.21.15.121:3300/0 conn(0x7fd22415f400 0x7fd22406eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.175+0000 7fd22ba72640 1 --2- 172.21.15.77:0/2222532904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd224143940 0x7fd22406e650 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.175+0000 7fd22ba72640 1 -- 172.21.15.77:0/2222532904 >> 172.21.15.77:0/2222532904 conn(0x7fd2240b1820 msgr2=0x7fd2240b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.175+0000 7fd22ba72640 1 -- 172.21.15.77:0/2222532904 shutdown_connections 2024-04-25T01:24:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.175+0000 7fd22ba72640 1 -- 172.21.15.77:0/2222532904 wait complete. 2024-04-25T01:24:08.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.176+0000 7fd22ba72640 1 Processor -- start 2024-04-25T01:24:08.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.176+0000 7fd22ba72640 1 -- start start 2024-04-25T01:24:08.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22ba72640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd224143940 0x7fd22413ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22ba72640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd224143d20 0x7fd22413f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22ba72640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd22415f400 0x7fd22413c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22ba72640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd22406d910 con 0x7fd22415f400 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22ba72640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd22406d790 con 0x7fd224143d20 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22ba72640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd22406da90 con 0x7fd224143940 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22a00f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd22415f400 0x7fd22413c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22a00f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd22415f400 0x7fd22413c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48462/0 (socket says 172.21.15.77:48462) 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22a00f640 1 -- 172.21.15.77:0/3701025274 learned_addr learned my addr 172.21.15.77:0/3701025274 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22980e640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd224143940 0x7fd22413ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22900d640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd224143d20 0x7fd22413f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22a00f640 1 -- 172.21.15.77:0/3701025274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd224143940 msgr2=0x7fd22413ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22a00f640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd224143940 0x7fd22413ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22a00f640 1 -- 172.21.15.77:0/3701025274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd224143d20 msgr2=0x7fd22413f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22a00f640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd224143d20 0x7fd22413f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22a00f640 1 -- 172.21.15.77:0/3701025274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2240ce770 con 0x7fd22415f400 2024-04-25T01:24:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22980e640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd224143940 0x7fd22413ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22900d640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd224143d20 0x7fd22413f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22a00f640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd22415f400 0x7fd22413c6b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd21c00b410 tx=0x7fd21c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd21a7fc640 1 -- 172.21.15.77:0/3701025274 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21c015020 con 0x7fd22415f400 2024-04-25T01:24:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd21a7fc640 1 -- 172.21.15.77:0/3701025274 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd21c011850 con 0x7fd22415f400 2024-04-25T01:24:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd21a7fc640 1 -- 172.21.15.77:0/3701025274 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21c011c40 con 0x7fd22415f400 2024-04-25T01:24:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.177+0000 7fd22ba72640 1 -- 172.21.15.77:0/3701025274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2240cea50 con 0x7fd22415f400 2024-04-25T01:24:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.178+0000 7fd22ba72640 1 -- 172.21.15.77:0/3701025274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd224153840 con 0x7fd22415f400 2024-04-25T01:24:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.179+0000 7fd22ba72640 1 -- 172.21.15.77:0/3701025274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd224068fb0 con 0x7fd22415f400 2024-04-25T01:24:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.179+0000 7fd21a7fc640 1 -- 172.21.15.77:0/3701025274 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd21c018020 con 0x7fd22415f400 2024-04-25T01:24:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.179+0000 7fd21a7fc640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd20806be30 0x7fd20806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.179+0000 7fd21a7fc640 1 -- 172.21.15.77:0/3701025274 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7fd21c08afa0 con 0x7fd22415f400 2024-04-25T01:24:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.179+0000 7fd22980e640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd20806be30 0x7fd20806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.180+0000 7fd22980e640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd20806be30 0x7fd20806e2f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd220033000 tx=0x7fd220009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:08.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.184+0000 7fd21a7fc640 1 -- 172.21.15.77:0/3701025274 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd21c01f020 con 0x7fd22415f400 2024-04-25T01:24:08.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.368+0000 7fd22ba72640 1 -- 172.21.15.77:0/3701025274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd2240cd760 con 0x7fd22415f400 2024-04-25T01:24:08.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.369+0000 7fd21a7fc640 1 -- 172.21.15.77:0/3701025274 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v55) ==== 80+0+246 (secure 0 0 0) 0x7fd21c059710 con 0x7fd22415f400 2024-04-25T01:24:08.349 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6836/3189776231 2024-04-26T01:24:01.947151+0000 2024-04-25T01:24:08.349 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6834/3189776231 2024-04-26T01:24:01.947151+0000 2024-04-25T01:24:08.349 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.121:6834/3834135103 2024-04-26T01:24:00.948815+0000 2024-04-25T01:24:08.349 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.121:6832/3834135103 2024-04-26T01:24:00.948815+0000 2024-04-25T01:24:08.349 INFO:teuthology.orchestra.run.smithi077.stderr:listed 4 entries 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 -- 172.21.15.77:0/3701025274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd20806be30 msgr2=0x7fd20806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd20806be30 0x7fd20806e2f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd220033000 tx=0x7fd220009000 comp rx=0 tx=0).stop 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 -- 172.21.15.77:0/3701025274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd22415f400 msgr2=0x7fd22413c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd22415f400 0x7fd22413c6b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd21c00b410 tx=0x7fd21c00b8d0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 -- 172.21.15.77:0/3701025274 shutdown_connections 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fd20806be30 0x7fd20806e2f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd22415f400 0x7fd22413c6b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd224143d20 0x7fd22413f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 --2- 172.21.15.77:0/3701025274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd224143940 0x7fd22413ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 -- 172.21.15.77:0/3701025274 >> 172.21.15.77:0/3701025274 conn(0x7fd2240b1820 msgr2=0x7fd2240afbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 -- 172.21.15.77:0/3701025274 shutdown_connections 2024-04-25T01:24:08.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.371+0000 7fd22ba72640 1 -- 172.21.15.77:0/3701025274 wait complete. 2024-04-25T01:24:08.361 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T01:24:08.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:08.361 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T01:24:08.388 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:24:08.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:08.388 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T01:24:08.443 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:24:08.443 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:24:08.443 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:24:08.462 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=434 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:24:08.462 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:24:08.462 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:24:08.462 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:24:08.498 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T01:24:08.499 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:24:08.499 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:24:08.499 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:24:08.520 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=434 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:24:08.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:08.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.655+0000 7fee09116640 1 Processor -- start 2024-04-25T01:24:08.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.655+0000 7fee09116640 1 -- start start 2024-04-25T01:24:08.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.656+0000 7fee09116640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fee04143d20 0x7fee0415ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.656+0000 7fee09116640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 0x7fee0415f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.656+0000 7fee09116640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee0415f770 con 0x7fee04143940 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.656+0000 7fee09116640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fee04070e60 con 0x7fee04143d20 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.656+0000 7fee09116640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fee04071140 con 0x7fee0406ed60 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.656+0000 7fee0359e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 0x7fee0415f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.656+0000 7fee0259c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fee04143d20 0x7fee0415ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.656+0000 7fee0359e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 0x7fee0415f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:38214/0 (socket says 172.21.15.77:38214) 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.656+0000 7fee0359e640 1 -- 172.21.15.77:0/2061065912 learned_addr learned my addr 172.21.15.77:0/2061065912 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.656+0000 7fee02d9d640 1 --1- 172.21.15.77:0/2061065912 >> v1:172.21.15.121:6789/0 conn(0x7fee04143940 0x7fee0406e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:55808/0 (socket says 172.21.15.77:55808) 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.657+0000 7fee0259c640 1 -- 172.21.15.77:0/2061065912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 msgr2=0x7fee0415f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.657+0000 7fee0259c640 1 --2- 172.21.15.77:0/2061065912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 0x7fee0415f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.657+0000 7fee0259c640 1 -- 172.21.15.77:0/2061065912 >> v1:172.21.15.121:6789/0 conn(0x7fee04143940 legacy=0x7fee0406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.657+0000 7fee0259c640 1 -- 172.21.15.77:0/2061065912 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee04071420 con 0x7fee04143d20 2024-04-25T01:24:08.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.657+0000 7fee01d9b640 1 -- 172.21.15.77:0/2061065912 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1054102660 0 0) 0x7fee0415f770 con 0x7fee04143940 2024-04-25T01:24:08.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.657+0000 7fee0359e640 1 --2- 172.21.15.77:0/2061065912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 0x7fee0415f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:08.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.657+0000 7fee0259c640 1 --2- 172.21.15.77:0/2061065912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee04143d20 0x7fee0415ecf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fedec0097d0 tx=0x7fedec02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24d87248e90ccb8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:08.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.657+0000 7fee01d9b640 1 -- 172.21.15.77:0/2061065912 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec037070 con 0x7fee04143d20 2024-04-25T01:24:08.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.658+0000 7fee01d9b640 1 -- 172.21.15.77:0/2061065912 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedec032440 con 0x7fee04143d20 2024-04-25T01:24:08.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.658+0000 7fee01d9b640 1 -- 172.21.15.77:0/2061065912 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec032810 con 0x7fee04143d20 2024-04-25T01:24:08.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.658+0000 7fee09116640 1 -- 172.21.15.77:0/2061065912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee04143d20 msgr2=0x7fee0415ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.658+0000 7fee09116640 1 --2- 172.21.15.77:0/2061065912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee04143d20 0x7fee0415ecf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fedec0097d0 tx=0x7fedec02a1c0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.658+0000 7fee09116640 1 -- 172.21.15.77:0/2061065912 shutdown_connections 2024-04-25T01:24:08.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.658+0000 7fee09116640 1 --2- 172.21.15.77:0/2061065912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 0x7fee0415f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.658+0000 7fee09116640 1 --2- 172.21.15.77:0/2061065912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee04143d20 0x7fee0415ecf0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.658+0000 7fee09116640 1 -- 172.21.15.77:0/2061065912 >> 172.21.15.77:0/2061065912 conn(0x7fee040b1820 msgr2=0x7fee040b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:08.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.658+0000 7fee09116640 1 -- 172.21.15.77:0/2061065912 shutdown_connections 2024-04-25T01:24:08.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.658+0000 7fee09116640 1 -- 172.21.15.77:0/2061065912 wait complete. 2024-04-25T01:24:08.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.659+0000 7fee09116640 1 Processor -- start 2024-04-25T01:24:08.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee09116640 1 -- start start 2024-04-25T01:24:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee09116640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 0x7fee0413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee09116640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee04143940 0x7fee040cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee09116640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee04143d20 0x7fee0415a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee09116640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fee04071da0 con 0x7fee04143940 2024-04-25T01:24:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee09116640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fee04071c20 con 0x7fee04143d20 2024-04-25T01:24:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee09116640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fee04071f20 con 0x7fee0406ed60 2024-04-25T01:24:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee0259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee04143940 0x7fee040cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee0259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee04143940 0x7fee040cd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48466/0 (socket says 172.21.15.77:48466) 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee0259c640 1 -- 172.21.15.77:0/4178802993 learned_addr learned my addr 172.21.15.77:0/4178802993 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee0359e640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee04143d20 0x7fee0415a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.660+0000 7fee02d9d640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 0x7fee0413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.661+0000 7fee0259c640 1 -- 172.21.15.77:0/4178802993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 msgr2=0x7fee0413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.661+0000 7fee0259c640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 0x7fee0413f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.661+0000 7fee0259c640 1 -- 172.21.15.77:0/4178802993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee04143d20 msgr2=0x7fee0415a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.661+0000 7fee0259c640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee04143d20 0x7fee0415a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.661+0000 7fee0259c640 1 -- 172.21.15.77:0/4178802993 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0415afb0 con 0x7fee04143940 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.661+0000 7fee0259c640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee04143940 0x7fee040cd990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fedec0098f0 tx=0x7fedec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.661+0000 7fedfb7fe640 1 -- 172.21.15.77:0/4178802993 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec037050 con 0x7fee04143940 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.661+0000 7fedfb7fe640 1 -- 172.21.15.77:0/4178802993 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedec038040 con 0x7fee04143940 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.661+0000 7fee09116640 1 -- 172.21.15.77:0/4178802993 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0415b290 con 0x7fee04143940 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.661+0000 7fedfb7fe640 1 -- 172.21.15.77:0/4178802993 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec004490 con 0x7fee04143940 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.661+0000 7fee09116640 1 -- 172.21.15.77:0/4178802993 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee040cd470 con 0x7fee04143940 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.662+0000 7fee09116640 1 -- 172.21.15.77:0/4178802993 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee04068fb0 con 0x7fee04143940 2024-04-25T01:24:08.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.662+0000 7fedfb7fe640 1 -- 172.21.15.77:0/4178802993 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fedec03c6d0 con 0x7fee04143940 2024-04-25T01:24:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.663+0000 7fedfb7fe640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fede406be30 0x7fede406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.663+0000 7fedfb7fe640 1 -- 172.21.15.77:0/4178802993 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7fedec0ab9a0 con 0x7fee04143940 2024-04-25T01:24:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.663+0000 7fee02d9d640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fede406be30 0x7fede406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.663+0000 7fee02d9d640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fede406be30 0x7fede406e2f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0012e0 tx=0x7fedfc001700 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:08.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.667+0000 7fedfb7fe640 1 -- 172.21.15.77:0/4178802993 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fedec079a40 con 0x7fee04143940 2024-04-25T01:24:08.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.881+0000 7fee09116640 1 -- 172.21.15.77:0/4178802993 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee0406d150 con 0x7fee04143940 2024-04-25T01:24:08.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.881+0000 7fedfb7fe640 1 -- 172.21.15.77:0/4178802993 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5188 (secure 0 0 0) 0x7fedec07dd40 con 0x7fee04143940 2024-04-25T01:24:08.861 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:08.861 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5129,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1374960539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1374960539},{"type":"v1","addr":"172.21.15.121:6834","nonce":1374960539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5467,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3910406676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":3910406676},{"type":"v1","addr":"172.21.15.121:6835","nonce":3910406676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6120,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3088617511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3088617511},{"type":"v1","addr":"172.21.15.77:6837","nonce":3088617511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:23:28.170422+0000","modified":"2024-04-25T01:24:04.977155+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/2057169374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2057169374},{"type":"v1","addr":"172.21.15.77:6835","nonce":2057169374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6102,"qdb_cluster":[6102]},"id":3}]} 2024-04-25T01:24:08.862 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 36 2024-04-25T01:24:08.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 -- 172.21.15.77:0/4178802993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fede406be30 msgr2=0x7fede406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fede406be30 0x7fede406e2f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0012e0 tx=0x7fedfc001700 comp rx=0 tx=0).stop 2024-04-25T01:24:08.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 -- 172.21.15.77:0/4178802993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee04143940 msgr2=0x7fee040cd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee04143940 0x7fee040cd990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fedec0098f0 tx=0x7fedec0046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 -- 172.21.15.77:0/4178802993 shutdown_connections 2024-04-25T01:24:08.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fede406be30 0x7fede406e2f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee04143940 0x7fee040cd990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee04143d20 0x7fee0415a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 --2- 172.21.15.77:0/4178802993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee0406ed60 0x7fee0413f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 -- 172.21.15.77:0/4178802993 >> 172.21.15.77:0/4178802993 conn(0x7fee040b1820 msgr2=0x7fee0413fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:08.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 -- 172.21.15.77:0/4178802993 shutdown_connections 2024-04-25T01:24:08.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.883+0000 7fee09116640 1 -- 172.21.15.77:0/4178802993 wait complete. 2024-04-25T01:24:08.874 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:24:08.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.991+0000 7f33b9921640 1 Processor -- start 2024-04-25T01:24:08.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.991+0000 7f33b9921640 1 -- start start 2024-04-25T01:24:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.992+0000 7f33b9921640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33b415e4c0 0x7f33b40d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.992+0000 7f33b9921640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b40d1e90 con 0x7f33b40d2d40 2024-04-25T01:24:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.992+0000 7f33b9921640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b40b54a0 con 0x7f33b40d4a10 2024-04-25T01:24:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.992+0000 7f33b9921640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f33b40d2130 con 0x7f33b415e4c0 2024-04-25T01:24:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.992+0000 7f33b37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f33b40d2d40 0x7f33b40d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49152/0 (socket says 172.21.15.77:49152) 2024-04-25T01:24:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.992+0000 7f33b37fe640 1 -- 172.21.15.77:0/2926957561 learned_addr learned my addr 172.21.15.77:0/2926957561 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.993+0000 7f33b2ffd640 1 --2- 172.21.15.77:0/2926957561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33b415e4c0 0x7f33b40d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.993+0000 7f33b1ffb640 1 -- 172.21.15.77:0/2926957561 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 631190554 0 0) 0x7f33b40d1e90 con 0x7f33b40d2d40 2024-04-25T01:24:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.993+0000 7f33b1ffb640 1 -- 172.21.15.77:0/2926957561 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f339c003520 con 0x7f33b40d2d40 2024-04-25T01:24:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.993+0000 7f33b1ffb640 1 -- 172.21.15.77:0/2926957561 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3756079247 0 0) 0x7f33b40b54a0 con 0x7f33b40d4a10 2024-04-25T01:24:08.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.993+0000 7f33b1ffb640 1 -- 172.21.15.77:0/2926957561 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33b40d1e90 con 0x7f33b40d4a10 2024-04-25T01:24:08.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.993+0000 7f33b2ffd640 1 -- 172.21.15.77:0/2926957561 >> v1:172.21.15.121:6789/0 conn(0x7f33b40d4a10 legacy=0x7f33b40b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.993+0000 7f33b2ffd640 1 -- 172.21.15.77:0/2926957561 >> v1:172.21.15.77:6789/0 conn(0x7f33b40d2d40 legacy=0x7f33b40d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.993+0000 7f33b1ffb640 1 -- 172.21.15.77:0/2926957561 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2732329158 0 0) 0x7f339c003520 con 0x7f33b40d2d40 2024-04-25T01:24:08.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.993+0000 7f33b2ffd640 1 -- 172.21.15.77:0/2926957561 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b40b5780 con 0x7f33b415e4c0 2024-04-25T01:24:08.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.994+0000 7f33b2ffd640 1 --2- 172.21.15.77:0/2926957561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33b415e4c0 0x7f33b40d2800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009dc0 tx=0x7f33a402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc6762cbdd73c16a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:08.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.994+0000 7f33b1ffb640 1 -- 172.21.15.77:0/2926957561 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac001670 con 0x7f33b415e4c0 2024-04-25T01:24:08.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.994+0000 7f33b1ffb640 1 -- 172.21.15.77:0/2926957561 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a4007540 con 0x7f33b415e4c0 2024-04-25T01:24:08.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.994+0000 7f33b1ffb640 1 -- 172.21.15.77:0/2926957561 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a40078f0 con 0x7f33b415e4c0 2024-04-25T01:24:08.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.994+0000 7f33b9921640 1 -- 172.21.15.77:0/2926957561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33b415e4c0 msgr2=0x7f33b40d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.994+0000 7f33b9921640 1 --2- 172.21.15.77:0/2926957561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33b415e4c0 0x7f33b40d2800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009dc0 tx=0x7f33a402a5d0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.994+0000 7f33b9921640 1 -- 172.21.15.77:0/2926957561 shutdown_connections 2024-04-25T01:24:08.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.994+0000 7f33b9921640 1 --2- 172.21.15.77:0/2926957561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33b415e4c0 0x7f33b40d2800 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.994+0000 7f33b9921640 1 -- 172.21.15.77:0/2926957561 >> 172.21.15.77:0/2926957561 conn(0x7f33b40d3ef0 msgr2=0x7f33b40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:08.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.994+0000 7f33b9921640 1 -- 172.21.15.77:0/2926957561 shutdown_connections 2024-04-25T01:24:08.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.994+0000 7f33b9921640 1 -- 172.21.15.77:0/2926957561 wait complete. 2024-04-25T01:24:08.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.996+0000 7f33b9921640 1 Processor -- start 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.996+0000 7f33b9921640 1 -- start start 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.996+0000 7f33b9921640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40d2d40 0x7f33b40c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.996+0000 7f33b9921640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33b40d4a10 0x7f33b40b7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.996+0000 7f33b9921640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33b415e4c0 0x7f33b4158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.996+0000 7f33b9921640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f33b415ea20 con 0x7f33b40d2d40 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.996+0000 7f33b9921640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f33b415e8a0 con 0x7f33b415e4c0 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.996+0000 7f33b9921640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f33b40b45f0 con 0x7f33b40d4a10 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40d2d40 0x7f33b40c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40d2d40 0x7f33b40c8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48480/0 (socket says 172.21.15.77:48480) 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b2ffd640 1 -- 172.21.15.77:0/3428487264 learned_addr learned my addr 172.21.15.77:0/3428487264 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b27fc640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33b40d4a10 0x7f33b40b7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b37fe640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33b415e4c0 0x7f33b4158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b2ffd640 1 -- 172.21.15.77:0/3428487264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33b40d4a10 msgr2=0x7f33b40b7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b2ffd640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33b40d4a10 0x7f33b40b7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b2ffd640 1 -- 172.21.15.77:0/3428487264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33b415e4c0 msgr2=0x7f33b4158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:08.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b2ffd640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33b415e4c0 0x7f33b4158010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:08.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b2ffd640 1 -- 172.21.15.77:0/3428487264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b40c8e60 con 0x7f33b40d2d40 2024-04-25T01:24:08.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b2ffd640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40d2d40 0x7f33b40c8780 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009ee0 tx=0x7f33a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:08.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f339bfff640 1 -- 172.21.15.77:0/3428487264 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a403f070 con 0x7f33b40d2d40 2024-04-25T01:24:08.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f339bfff640 1 -- 172.21.15.77:0/3428487264 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a4031bc0 con 0x7f33b40d2d40 2024-04-25T01:24:08.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b9921640 1 -- 172.21.15.77:0/3428487264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33b41f75f0 con 0x7f33b40d2d40 2024-04-25T01:24:08.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f339bfff640 1 -- 172.21.15.77:0/3428487264 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4009450 con 0x7f33b40d2d40 2024-04-25T01:24:08.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.997+0000 7f33b9921640 1 -- 172.21.15.77:0/3428487264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33b41f7bb0 con 0x7f33b40d2d40 2024-04-25T01:24:08.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.998+0000 7f33b9921640 1 -- 172.21.15.77:0/3428487264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33b4068fb0 con 0x7f33b40d2d40 2024-04-25T01:24:08.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.999+0000 7f339bfff640 1 -- 172.21.15.77:0/3428487264 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f33a403b410 con 0x7f33b40d2d40 2024-04-25T01:24:08.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.999+0000 7f339bfff640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f33a006be30 0x7f33a006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:08.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.999+0000 7f339bfff640 1 -- 172.21.15.77:0/3428487264 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f33a40ac620 con 0x7f33b40d2d40 2024-04-25T01:24:08.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:08.999+0000 7f33b27fc640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f33a006be30 0x7f33a006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:08.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.000+0000 7f33b27fc640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f33a006be30 0x7f33a006e2f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f33b40d4990 tx=0x7f33a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:08.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.005+0000 7f339bfff640 1 -- 172.21.15.77:0/3428487264 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f33a407a740 con 0x7f33b40d2d40 2024-04-25T01:24:09.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.177+0000 7f33b9921640 1 -- 172.21.15.77:0/3428487264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f33b40d46e0 con 0x7f33b40d2d40 2024-04-25T01:24:09.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.178+0000 7f339bfff640 1 -- 172.21.15.77:0/3428487264 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+17203 (secure 0 0 0) 0x7f33a407ea40 con 0x7f33b40d2d40 2024-04-25T01:24:09.159 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:09.159 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":55,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:24:01.947173+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:23:26.392274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T01:23:27.146093+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6836/3189776231":"2024-04-26T01:24:01.947151+0000","172.21.15.77:6834/3189776231":"2024-04-26T01:24:01.947151+0000","172.21.15.121:6834/3834135103":"2024-04-26T01:24:00.948815+0000","172.21.15.121:6832/3834135103":"2024-04-26T01:24:00.948815+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:24:09.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 -- 172.21.15.77:0/3428487264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f33a006be30 msgr2=0x7f33a006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f33a006be30 0x7f33a006e2f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f33b40d4990 tx=0x7f33a8009040 comp rx=0 tx=0).stop 2024-04-25T01:24:09.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 -- 172.21.15.77:0/3428487264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40d2d40 msgr2=0x7f33b40c8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40d2d40 0x7f33b40c8780 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009ee0 tx=0x7f33a4004a40 comp rx=0 tx=0).stop 2024-04-25T01:24:09.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 -- 172.21.15.77:0/3428487264 shutdown_connections 2024-04-25T01:24:09.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f33a006be30 0x7f33a006e2f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33b415e4c0 0x7f33b4158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33b40d4a10 0x7f33b40b7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 --2- 172.21.15.77:0/3428487264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40d2d40 0x7f33b40c8780 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 -- 172.21.15.77:0/3428487264 >> 172.21.15.77:0/3428487264 conn(0x7f33b40d3ef0 msgr2=0x7f33b415d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:09.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 -- 172.21.15.77:0/3428487264 shutdown_connections 2024-04-25T01:24:09.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.180+0000 7f33b9921640 1 -- 172.21.15.77:0/3428487264 wait complete. 2024-04-25T01:24:09.172 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:21:33.186152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:23:26.392274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:23:27.146093+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:24:09.172 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:24:09.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.291+0000 7f550b79d640 1 Processor -- start 2024-04-25T01:24:09.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.292+0000 7f550b79d640 1 -- start start 2024-04-25T01:24:09.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.292+0000 7f550b79d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f55040d23d0 0x7f55040d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.292+0000 7f550b79d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55040d27b0 0x7f55040d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.292+0000 7f550b79d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55040cea40 con 0x7f55040b9380 2024-04-25T01:24:09.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.292+0000 7f550b79d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f55040cece0 con 0x7f55040d23d0 2024-04-25T01:24:09.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.292+0000 7f550b79d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f55040cefc0 con 0x7f55040d27b0 2024-04-25T01:24:09.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.292+0000 7f5509539640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f55040d23d0 0x7f55040d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.292+0000 7f5509d3a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f55040b9380 0x7f55040b97f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49166/0 (socket says 172.21.15.77:49166) 2024-04-25T01:24:09.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.292+0000 7f5509d3a640 1 -- 172.21.15.77:0/3152881962 learned_addr learned my addr 172.21.15.77:0/3152881962 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:09.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.293+0000 7f5508d38640 1 --2- 172.21.15.77:0/3152881962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55040d27b0 0x7f55040d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.293+0000 7f5509539640 1 -- 172.21.15.77:0/3152881962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55040d27b0 msgr2=0x7f55040d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:09.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.293+0000 7f5509539640 1 --2- 172.21.15.77:0/3152881962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55040d27b0 0x7f55040d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.293+0000 7f5509539640 1 -- 172.21.15.77:0/3152881962 >> v1:172.21.15.77:6789/0 conn(0x7f55040b9380 legacy=0x7f55040b97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.293+0000 7f54fbfff640 1 -- 172.21.15.77:0/3152881962 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3819632254 0 0) 0x7f55040cea40 con 0x7f55040b9380 2024-04-25T01:24:09.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.293+0000 7f5509539640 1 -- 172.21.15.77:0/3152881962 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55040cf2a0 con 0x7f55040d23d0 2024-04-25T01:24:09.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.293+0000 7f5508d38640 1 --2- 172.21.15.77:0/3152881962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55040d27b0 0x7f55040d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:09.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.293+0000 7f5509539640 1 --2- 172.21.15.77:0/3152881962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55040d23d0 0x7f55040d4920 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5500009d90 tx=0x7f550002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77c7bc45d546fc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:09.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.294+0000 7f54fbfff640 1 -- 172.21.15.77:0/3152881962 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5500007740 con 0x7f55040d23d0 2024-04-25T01:24:09.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.294+0000 7f54fbfff640 1 -- 172.21.15.77:0/3152881962 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55000078e0 con 0x7f55040d23d0 2024-04-25T01:24:09.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.294+0000 7f54fbfff640 1 -- 172.21.15.77:0/3152881962 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5500007cb0 con 0x7f55040d23d0 2024-04-25T01:24:09.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.294+0000 7f550b79d640 1 -- 172.21.15.77:0/3152881962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55040d23d0 msgr2=0x7f55040d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.294+0000 7f550b79d640 1 --2- 172.21.15.77:0/3152881962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55040d23d0 0x7f55040d4920 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5500009d90 tx=0x7f550002a6f0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.294+0000 7f550b79d640 1 -- 172.21.15.77:0/3152881962 shutdown_connections 2024-04-25T01:24:09.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.294+0000 7f550b79d640 1 --2- 172.21.15.77:0/3152881962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55040d27b0 0x7f55040d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.294+0000 7f550b79d640 1 --2- 172.21.15.77:0/3152881962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55040d23d0 0x7f55040d4920 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.294+0000 7f550b79d640 1 -- 172.21.15.77:0/3152881962 >> 172.21.15.77:0/3152881962 conn(0x7f55040d3e30 msgr2=0x7f55040d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:09.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.294+0000 7f550b79d640 1 -- 172.21.15.77:0/3152881962 shutdown_connections 2024-04-25T01:24:09.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.294+0000 7f550b79d640 1 -- 172.21.15.77:0/3152881962 wait complete. 2024-04-25T01:24:09.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.295+0000 7f550b79d640 1 Processor -- start 2024-04-25T01:24:09.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.295+0000 7f550b79d640 1 -- start start 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f550b79d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55040b9380 0x7f55041503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f550b79d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55040d23d0 0x7f55041f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f550b79d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55040d27b0 0x7f55040c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f550b79d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f55040cfd80 con 0x7f55040b9380 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f550b79d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f55040cf860 con 0x7f55040d23d0 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f550b79d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f55040b6c80 con 0x7f55040d27b0 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f5509539640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55040b9380 0x7f55041503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f5509539640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55040b9380 0x7f55041503a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48490/0 (socket says 172.21.15.77:48490) 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f5509539640 1 -- 172.21.15.77:0/3688297199 learned_addr learned my addr 172.21.15.77:0/3688297199 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f5509d3a640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55040d27b0 0x7f55040c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f5508d38640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55040d23d0 0x7f55041f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f5509539640 1 -- 172.21.15.77:0/3688297199 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55040d27b0 msgr2=0x7f55040c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f5509539640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55040d27b0 0x7f55040c8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f5509539640 1 -- 172.21.15.77:0/3688297199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55040d23d0 msgr2=0x7f55041f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f5509539640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55040d23d0 0x7f55041f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f5509539640 1 -- 172.21.15.77:0/3688297199 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55041f3980 con 0x7f55040b9380 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.296+0000 7f5508d38640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55040d23d0 0x7f55041f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.297+0000 7f5509539640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55040b9380 0x7f55041503a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5500009d90 tx=0x7f5500004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.297+0000 7f54f9ffb640 1 -- 172.21.15.77:0/3688297199 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5500040070 con 0x7f55040b9380 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.297+0000 7f54f9ffb640 1 -- 172.21.15.77:0/3688297199 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5500031c40 con 0x7f55040b9380 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.297+0000 7f54f9ffb640 1 -- 172.21.15.77:0/3688297199 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550003c420 con 0x7f55040b9380 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.297+0000 7f550b79d640 1 -- 172.21.15.77:0/3688297199 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55040cb0a0 con 0x7f55040b9380 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.297+0000 7f550b79d640 1 -- 172.21.15.77:0/3688297199 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55040cb660 con 0x7f55040b9380 2024-04-25T01:24:09.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.298+0000 7f550b79d640 1 -- 172.21.15.77:0/3688297199 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5504068fb0 con 0x7f55040b9380 2024-04-25T01:24:09.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.298+0000 7f54f9ffb640 1 -- 172.21.15.77:0/3688297199 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5500039040 con 0x7f55040b9380 2024-04-25T01:24:09.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.299+0000 7f54f9ffb640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f54e806be30 0x7f54e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.299+0000 7f54f9ffb640 1 -- 172.21.15.77:0/3688297199 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f55000ac630 con 0x7f55040b9380 2024-04-25T01:24:09.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.299+0000 7f5508d38640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f54e806be30 0x7f54e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.299+0000 7f5508d38640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f54e806be30 0x7f54e806e2f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f54f4006fd0 tx=0x7f54f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:09.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.303+0000 7f54f9ffb640 1 -- 172.21.15.77:0/3688297199 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f550007ada0 con 0x7f55040b9380 2024-04-25T01:24:09.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.473+0000 7f550b79d640 1 -- 172.21.15.77:0/3688297199 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f550408df50 con 0x7f55040b9380 2024-04-25T01:24:09.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.474+0000 7f54f9ffb640 1 -- 172.21.15.77:0/3688297199 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) ==== 93+0+10 (secure 0 0 0) 0x7f550007a6a0 con 0x7f55040b9380 2024-04-25T01:24:09.455 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-04-25T01:24:09.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 -- 172.21.15.77:0/3688297199 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f54e806be30 msgr2=0x7f54e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f54e806be30 0x7f54e806e2f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f54f4006fd0 tx=0x7f54f4008040 comp rx=0 tx=0).stop 2024-04-25T01:24:09.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 -- 172.21.15.77:0/3688297199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55040b9380 msgr2=0x7f55041503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55040b9380 0x7f55041503a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5500009d90 tx=0x7f5500004a40 comp rx=0 tx=0).stop 2024-04-25T01:24:09.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 -- 172.21.15.77:0/3688297199 shutdown_connections 2024-04-25T01:24:09.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f54e806be30 0x7f54e806e2f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55040d27b0 0x7f55040c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55040d23d0 0x7f55041f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 --2- 172.21.15.77:0/3688297199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55040b9380 0x7f55041503a0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 -- 172.21.15.77:0/3688297199 >> 172.21.15.77:0/3688297199 conn(0x7f55040d3e30 msgr2=0x7f55040b85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:09.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 -- 172.21.15.77:0/3688297199 shutdown_connections 2024-04-25T01:24:09.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.476+0000 7f550b79d640 1 -- 172.21.15.77:0/3688297199 wait complete. 2024-04-25T01:24:09.467 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:24:09.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.585+0000 7f6b40e6b640 1 Processor -- start 2024-04-25T01:24:09.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.585+0000 7f6b40e6b640 1 -- start start 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.586+0000 7f6b40e6b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b3c0b9a90 0x7f6b3c0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.586+0000 7f6b40e6b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b3c0d27b0 0x7f6b3c0d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.586+0000 7f6b40e6b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b3c0cea40 con 0x7f6b3c0d23d0 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.586+0000 7f6b40e6b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6b3c0cece0 con 0x7f6b3c0b9a90 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.586+0000 7f6b40e6b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6b3c0cefc0 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.587+0000 7f6b39d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b3c0b9a90 0x7f6b3c0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.587+0000 7f6b39d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b3c0b9a90 0x7f6b3c0d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48506/0 (socket says 172.21.15.77:48506) 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.587+0000 7f6b39d9b640 1 -- 172.21.15.77:0/844990043 learned_addr learned my addr 172.21.15.77:0/844990043 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.587+0000 7f6b3ad9d640 1 --2- 172.21.15.77:0/844990043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b3c0d27b0 0x7f6b3c0d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.587+0000 7f6b39d9b640 1 -- 172.21.15.77:0/844990043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b3c0d27b0 msgr2=0x7f6b3c0d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.587+0000 7f6b39d9b640 1 --2- 172.21.15.77:0/844990043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b3c0d27b0 0x7f6b3c0d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.587+0000 7f6b39d9b640 1 -- 172.21.15.77:0/844990043 >> v1:172.21.15.121:6789/0 conn(0x7f6b3c0d23d0 legacy=0x7f6b3c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.587+0000 7f6b39d9b640 1 -- 172.21.15.77:0/844990043 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3c0cf2a0 con 0x7f6b3c0b9a90 2024-04-25T01:24:09.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b39d9b640 1 --2- 172.21.15.77:0/844990043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b3c0b9a90 0x7f6b3c0d4920 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009870 tx=0x7f6b3002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a33ecaa25201281 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:09.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b3959a640 1 -- 172.21.15.77:0/844990043 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30037070 con 0x7f6b3c0b9a90 2024-04-25T01:24:09.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b3959a640 1 -- 172.21.15.77:0/844990043 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b3002ae50 con 0x7f6b3c0b9a90 2024-04-25T01:24:09.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b3959a640 1 -- 172.21.15.77:0/844990043 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30032860 con 0x7f6b3c0b9a90 2024-04-25T01:24:09.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b40e6b640 1 -- 172.21.15.77:0/844990043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b3c0b9a90 msgr2=0x7f6b3c0d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b40e6b640 1 --2- 172.21.15.77:0/844990043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b3c0b9a90 0x7f6b3c0d4920 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009870 tx=0x7f6b3002a300 comp rx=0 tx=0).stop 2024-04-25T01:24:09.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b40e6b640 1 -- 172.21.15.77:0/844990043 shutdown_connections 2024-04-25T01:24:09.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b40e6b640 1 --2- 172.21.15.77:0/844990043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b3c0d27b0 0x7f6b3c0d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b40e6b640 1 --2- 172.21.15.77:0/844990043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b3c0b9a90 0x7f6b3c0d4920 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b40e6b640 1 -- 172.21.15.77:0/844990043 >> 172.21.15.77:0/844990043 conn(0x7f6b3c0d3e30 msgr2=0x7f6b3c0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:09.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b40e6b640 1 -- 172.21.15.77:0/844990043 shutdown_connections 2024-04-25T01:24:09.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.588+0000 7f6b40e6b640 1 -- 172.21.15.77:0/844990043 wait complete. 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.590+0000 7f6b40e6b640 1 Processor -- start 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.590+0000 7f6b40e6b640 1 -- start start 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.590+0000 7f6b40e6b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b3c0b9a90 0x7f6b3c1f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.590+0000 7f6b40e6b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b3c0d23d0 0x7f6b3c160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.590+0000 7f6b40e6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b3c0d27b0 0x7f6b3c160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.590+0000 7f6b40e6b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c0cfd80 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.590+0000 7f6b40e6b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c0cf860 con 0x7f6b3c0b9a90 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.590+0000 7f6b40e6b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6b3c0b6c80 con 0x7f6b3c0d23d0 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b3ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b3c0d27b0 0x7f6b3c160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b3ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b3c0d27b0 0x7f6b3c160960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48520/0 (socket says 172.21.15.77:48520) 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b3ad9d640 1 -- 172.21.15.77:0/389552119 learned_addr learned my addr 172.21.15.77:0/389552119 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b39d9b640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b3c0d23d0 0x7f6b3c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b3a59c640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b3c0b9a90 0x7f6b3c1f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b3ad9d640 1 -- 172.21.15.77:0/389552119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b3c0d23d0 msgr2=0x7f6b3c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b3ad9d640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b3c0d23d0 0x7f6b3c160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b3ad9d640 1 -- 172.21.15.77:0/389552119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b3c0b9a90 msgr2=0x7f6b3c1f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b3ad9d640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b3c0b9a90 0x7f6b3c1f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b3ad9d640 1 -- 172.21.15.77:0/389552119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3c14a3f0 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b3ad9d640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b3c0d27b0 0x7f6b3c160960 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6b34007e60 tx=0x7f6b3400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b2affd640 1 -- 172.21.15.77:0/389552119 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34014420 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b2affd640 1 -- 172.21.15.77:0/389552119 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b34019070 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b40e6b640 1 -- 172.21.15.77:0/389552119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b3c14a6d0 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b2affd640 1 -- 172.21.15.77:0/389552119 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34014820 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.591+0000 7f6b40e6b640 1 -- 172.21.15.77:0/389552119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b3c14ac90 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.592+0000 7f6b40e6b640 1 -- 172.21.15.77:0/389552119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b3c068fb0 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.593+0000 7f6b2affd640 1 -- 172.21.15.77:0/389552119 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6b340149c0 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.593+0000 7f6b2affd640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b1c06be30 0x7f6b1c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.593+0000 7f6b3a59c640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b1c06be30 0x7f6b1c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.593+0000 7f6b2affd640 1 -- 172.21.15.77:0/389552119 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f6b3408b7b0 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.594+0000 7f6b3a59c640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b1c06be30 0x7f6b1c06e2f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c001780 tx=0x7f6b2c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:09.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.598+0000 7f6b2affd640 1 -- 172.21.15.77:0/389552119 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6b340598d0 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.774+0000 7f6b40e6b640 1 -- 172.21.15.77:0/389552119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6b3c08df50 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.775+0000 7f6b2affd640 1 -- 172.21.15.77:0/389552119 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) ==== 104+0+11 (secure 0 0 0) 0x7f6b3405dbd0 con 0x7f6b3c0d27b0 2024-04-25T01:24:09.756 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-04-25T01:24:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 -- 172.21.15.77:0/389552119 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b1c06be30 msgr2=0x7f6b1c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b1c06be30 0x7f6b1c06e2f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c001780 tx=0x7f6b2c009040 comp rx=0 tx=0).stop 2024-04-25T01:24:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 -- 172.21.15.77:0/389552119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b3c0d27b0 msgr2=0x7f6b3c160960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b3c0d27b0 0x7f6b3c160960 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6b34007e60 tx=0x7f6b3400e4b0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 -- 172.21.15.77:0/389552119 shutdown_connections 2024-04-25T01:24:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f6b1c06be30 0x7f6b1c06e2f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b3c0d27b0 0x7f6b3c160960 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b3c0d23d0 0x7f6b3c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 --2- 172.21.15.77:0/389552119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b3c0b9a90 0x7f6b3c1f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 -- 172.21.15.77:0/389552119 >> 172.21.15.77:0/389552119 conn(0x7f6b3c0d3e30 msgr2=0x7f6b3c0cd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:09.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 -- 172.21.15.77:0/389552119 shutdown_connections 2024-04-25T01:24:09.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.777+0000 7f6b40e6b640 1 -- 172.21.15.77:0/389552119 wait complete. 2024-04-25T01:24:09.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:24:09.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.887+0000 7f47354ca640 1 Processor -- start 2024-04-25T01:24:09.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.887+0000 7f47354ca640 1 -- start start 2024-04-25T01:24:09.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f47354ca640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4730158a70 0x7f47300b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f47354ca640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 0x7f47300b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f47354ca640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f473015c210 con 0x7f473015b520 2024-04-25T01:24:09.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f47354ca640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f47300b2ce0 con 0x7f4730158a70 2024-04-25T01:24:09.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f47354ca640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f47300b2fc0 con 0x7f47300b37c0 2024-04-25T01:24:09.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f4727fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 0x7f47300b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f472effd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4730158a70 0x7f47300b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f4727fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 0x7f47300b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.77:38296/0 (socket says 172.21.15.77:38296) 2024-04-25T01:24:09.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f4727fff640 1 -- 172.21.15.77:0/3759510421 learned_addr learned my addr 172.21.15.77:0/3759510421 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:09.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f472effd640 1 -- 172.21.15.77:0/3759510421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 msgr2=0x7f47300b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:09.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f472effd640 1 --2- 172.21.15.77:0/3759510421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 0x7f47300b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f472effd640 1 -- 172.21.15.77:0/3759510421 >> v1:172.21.15.121:6789/0 conn(0x7f473015b520 legacy=0x7f473015b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f472effd640 1 -- 172.21.15.77:0/3759510421 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47300b32a0 con 0x7f4730158a70 2024-04-25T01:24:09.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.888+0000 7f4727fff640 1 --2- 172.21.15.77:0/3759510421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 0x7f47300b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:09.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.889+0000 7f472effd640 1 --2- 172.21.15.77:0/3759510421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4730158a70 0x7f47300b40c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f47200097a0 tx=0x7f472002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efabfb05492e2812 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:09.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.889+0000 7f472e7fc640 1 -- 172.21.15.77:0/3759510421 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4720037070 con 0x7f4730158a70 2024-04-25T01:24:09.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.889+0000 7f472e7fc640 1 -- 172.21.15.77:0/3759510421 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4720032440 con 0x7f4730158a70 2024-04-25T01:24:09.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.889+0000 7f472e7fc640 1 -- 172.21.15.77:0/3759510421 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4720032810 con 0x7f4730158a70 2024-04-25T01:24:09.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.890+0000 7f47354ca640 1 -- 172.21.15.77:0/3759510421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4730158a70 msgr2=0x7f47300b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.890+0000 7f47354ca640 1 --2- 172.21.15.77:0/3759510421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4730158a70 0x7f47300b40c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f47200097a0 tx=0x7f472002a180 comp rx=0 tx=0).stop 2024-04-25T01:24:09.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.890+0000 7f47354ca640 1 -- 172.21.15.77:0/3759510421 shutdown_connections 2024-04-25T01:24:09.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.890+0000 7f47354ca640 1 --2- 172.21.15.77:0/3759510421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 0x7f47300b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.890+0000 7f47354ca640 1 --2- 172.21.15.77:0/3759510421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4730158a70 0x7f47300b40c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.890+0000 7f47354ca640 1 -- 172.21.15.77:0/3759510421 >> 172.21.15.77:0/3759510421 conn(0x7f473006b720 msgr2=0x7f47300b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:09.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.890+0000 7f47354ca640 1 -- 172.21.15.77:0/3759510421 shutdown_connections 2024-04-25T01:24:09.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.890+0000 7f47354ca640 1 -- 172.21.15.77:0/3759510421 wait complete. 2024-04-25T01:24:09.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.891+0000 7f47354ca640 1 Processor -- start 2024-04-25T01:24:09.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.891+0000 7f47354ca640 1 -- start start 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f47354ca640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 0x7f47301e4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f47354ca640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4730158a70 0x7f4730132390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f47354ca640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f473015b520 0x7f4730135600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f47354ca640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f473006cb70 con 0x7f473015b520 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f47354ca640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f473006c9f0 con 0x7f4730158a70 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f47354ca640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f473006ccf0 con 0x7f47300b37c0 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f472f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f473015b520 0x7f4730135600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f472f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f473015b520 0x7f4730135600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48534/0 (socket says 172.21.15.77:48534) 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f472f7fe640 1 -- 172.21.15.77:0/3529671711 learned_addr learned my addr 172.21.15.77:0/3529671711 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f4727fff640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4730158a70 0x7f4730132390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f472effd640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 0x7f47301e4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f472f7fe640 1 -- 172.21.15.77:0/3529671711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 msgr2=0x7f47301e4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f472f7fe640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 0x7f47301e4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f472f7fe640 1 -- 172.21.15.77:0/3529671711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4730158a70 msgr2=0x7f4730132390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:09.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f472f7fe640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4730158a70 0x7f4730132390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:09.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f472f7fe640 1 -- 172.21.15.77:0/3529671711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4730132be0 con 0x7f473015b520 2024-04-25T01:24:09.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f4727fff640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4730158a70 0x7f4730132390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:09.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f472f7fe640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f473015b520 0x7f4730135600 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f472800c730 tx=0x7f472800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:09.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.892+0000 7f47277fe640 1 -- 172.21.15.77:0/3529671711 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4728016020 con 0x7f473015b520 2024-04-25T01:24:09.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.893+0000 7f47277fe640 1 -- 172.21.15.77:0/3529671711 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4728006d50 con 0x7f473015b520 2024-04-25T01:24:09.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.893+0000 7f47354ca640 1 -- 172.21.15.77:0/3529671711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4730132ec0 con 0x7f473015b520 2024-04-25T01:24:09.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.893+0000 7f47277fe640 1 -- 172.21.15.77:0/3529671711 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4728003bb0 con 0x7f473015b520 2024-04-25T01:24:09.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.893+0000 7f47354ca640 1 -- 172.21.15.77:0/3529671711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4730133480 con 0x7f473015b520 2024-04-25T01:24:09.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.893+0000 7f47354ca640 1 -- 172.21.15.77:0/3529671711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4730133680 con 0x7f473015b520 2024-04-25T01:24:09.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.894+0000 7f47277fe640 1 -- 172.21.15.77:0/3529671711 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f47280050d0 con 0x7f473015b520 2024-04-25T01:24:09.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.894+0000 7f47277fe640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f471006be30 0x7f471006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:09.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.894+0000 7f47277fe640 1 -- 172.21.15.77:0/3529671711 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f472808acc0 con 0x7f473015b520 2024-04-25T01:24:09.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.894+0000 7f472effd640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f471006be30 0x7f471006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:09.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.895+0000 7f472effd640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f471006be30 0x7f471006e2f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4720034000 tx=0x7f472002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:09.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:09.899+0000 7f47277fe640 1 -- 172.21.15.77:0/3529671711 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4730133680 con 0x7f473015b520 2024-04-25T01:24:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.073+0000 7f47354ca640 1 -- 172.21.15.77:0/3529671711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4730068fb0 con 0x7f473015b520 2024-04-25T01:24:10.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.073+0000 7f47277fe640 1 -- 172.21.15.77:0/3529671711 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) ==== 100+0+11 (secure 0 0 0) 0x7f4728091080 con 0x7f473015b520 2024-04-25T01:24:10.054 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-04-25T01:24:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 -- 172.21.15.77:0/3529671711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f471006be30 msgr2=0x7f471006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f471006be30 0x7f471006e2f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4720034000 tx=0x7f472002e000 comp rx=0 tx=0).stop 2024-04-25T01:24:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 -- 172.21.15.77:0/3529671711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f473015b520 msgr2=0x7f4730135600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f473015b520 0x7f4730135600 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f472800c730 tx=0x7f472800cb30 comp rx=0 tx=0).stop 2024-04-25T01:24:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 -- 172.21.15.77:0/3529671711 shutdown_connections 2024-04-25T01:24:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f471006be30 0x7f471006e2f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f473015b520 0x7f4730135600 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4730158a70 0x7f4730132390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 --2- 172.21.15.77:0/3529671711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300b37c0 0x7f47301e4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 -- 172.21.15.77:0/3529671711 >> 172.21.15.77:0/3529671711 conn(0x7f473006b720 msgr2=0x7f47300b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 -- 172.21.15.77:0/3529671711 shutdown_connections 2024-04-25T01:24:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.075+0000 7f47354ca640 1 -- 172.21.15.77:0/3529671711 wait complete. 2024-04-25T01:24:10.067 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:10.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.183+0000 7f1fda190640 1 Processor -- start 2024-04-25T01:24:10.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.183+0000 7f1fda190640 1 -- start start 2024-04-25T01:24:10.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.184+0000 7f1fda190640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1fd415e4c0 0x7f1fd40d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.184+0000 7f1fda190640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fd40d4a10 0x7f1fd40b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.184+0000 7f1fda190640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1fd40d2d40 0x7f1fd40b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.184+0000 7f1fda190640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f1fd40d1e90 con 0x7f1fd40d2d40 2024-04-25T01:24:10.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.184+0000 7f1fda190640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1fd40b54a0 con 0x7f1fd415e4c0 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.184+0000 7f1fda190640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1fd40d2130 con 0x7f1fd40d4a10 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.184+0000 7f1fd3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1fd40d2d40 0x7f1fd40b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.184+0000 7f1fd3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1fd40d2d40 0x7f1fd40b4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48540/0 (socket says 172.21.15.77:48540) 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.184+0000 7f1fd3fff640 1 -- 172.21.15.77:0/2227226786 learned_addr learned my addr 172.21.15.77:0/2227226786 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.185+0000 7f1fd37fe640 1 --2- 172.21.15.77:0/2227226786 >> v2:172.21.15.121:3300/0 conn(0x7f1fd415e4c0 0x7f1fd40d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.185+0000 7f1fcaffd640 1 --2- 172.21.15.77:0/2227226786 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fd40d4a10 0x7f1fd40b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.185+0000 7f1fd3fff640 1 -- 172.21.15.77:0/2227226786 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fd40d4a10 msgr2=0x7f1fd40b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.185+0000 7f1fd3fff640 1 --2- 172.21.15.77:0/2227226786 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fd40d4a10 0x7f1fd40b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.185+0000 7f1fd3fff640 1 -- 172.21.15.77:0/2227226786 >> v2:172.21.15.121:3300/0 conn(0x7f1fd415e4c0 msgr2=0x7f1fd40d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.185+0000 7f1fd3fff640 1 --2- 172.21.15.77:0/2227226786 >> v2:172.21.15.121:3300/0 conn(0x7f1fd415e4c0 0x7f1fd40d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.185+0000 7f1fd3fff640 1 -- 172.21.15.77:0/2227226786 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd40b5780 con 0x7f1fd40d2d40 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.185+0000 7f1fcaffd640 1 --2- 172.21.15.77:0/2227226786 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fd40d4a10 0x7f1fd40b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.185+0000 7f1fd3fff640 1 --2- 172.21.15.77:0/2227226786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd40d2d40 0x7f1fd40b4420 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4067960 tx=0x7f1fcc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c16ab26d8231ecc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fd2ffd640 1 -- 172.21.15.77:0/2227226786 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc036070 con 0x7f1fd40d2d40 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fd2ffd640 1 -- 172.21.15.77:0/2227226786 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fcc031600 con 0x7f1fd40d2d40 2024-04-25T01:24:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fd2ffd640 1 -- 172.21.15.77:0/2227226786 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc0319d0 con 0x7f1fd40d2d40 2024-04-25T01:24:10.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fda190640 1 -- 172.21.15.77:0/2227226786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd40d2d40 msgr2=0x7f1fd40b4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fda190640 1 --2- 172.21.15.77:0/2227226786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd40d2d40 0x7f1fd40b4420 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4067960 tx=0x7f1fcc02a610 comp rx=0 tx=0).stop 2024-04-25T01:24:10.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fda190640 1 -- 172.21.15.77:0/2227226786 shutdown_connections 2024-04-25T01:24:10.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fda190640 1 --2- 172.21.15.77:0/2227226786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd40d2d40 0x7f1fd40b4420 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fda190640 1 --2- 172.21.15.77:0/2227226786 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fd40d4a10 0x7f1fd40b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fda190640 1 --2- 172.21.15.77:0/2227226786 >> v2:172.21.15.121:3300/0 conn(0x7f1fd415e4c0 0x7f1fd40d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fda190640 1 -- 172.21.15.77:0/2227226786 >> 172.21.15.77:0/2227226786 conn(0x7f1fd40d3ef0 msgr2=0x7f1fd40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:10.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fda190640 1 -- 172.21.15.77:0/2227226786 shutdown_connections 2024-04-25T01:24:10.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.186+0000 7f1fda190640 1 -- 172.21.15.77:0/2227226786 wait complete. 2024-04-25T01:24:10.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.187+0000 7f1fda190640 1 Processor -- start 2024-04-25T01:24:10.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.188+0000 7f1fda190640 1 -- start start 2024-04-25T01:24:10.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.188+0000 7f1fda190640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fd40d2d40 0x7f1fd40b7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.188+0000 7f1fda190640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd40d4a10 0x7f1fd41513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.188+0000 7f1fda190640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fd415e4c0 0x7f1fd40b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.188+0000 7f1fda190640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd415ea20 con 0x7f1fd40d4a10 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.188+0000 7f1fda190640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd415e8a0 con 0x7f1fd40d2d40 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.188+0000 7f1fda190640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1fd40d0900 con 0x7f1fd415e4c0 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.188+0000 7f1fcaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd40d4a10 0x7f1fd41513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.188+0000 7f1fcaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd40d4a10 0x7f1fd41513d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48554/0 (socket says 172.21.15.77:48554) 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.188+0000 7f1fcaffd640 1 -- 172.21.15.77:0/1414040269 learned_addr learned my addr 172.21.15.77:0/1414040269 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fd3fff640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fd415e4c0 0x7f1fd40b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fd37fe640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fd40d2d40 0x7f1fd40b7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fcaffd640 1 -- 172.21.15.77:0/1414040269 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fd415e4c0 msgr2=0x7f1fd40b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fcaffd640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fd415e4c0 0x7f1fd40b2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fcaffd640 1 -- 172.21.15.77:0/1414040269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fd40d2d40 msgr2=0x7f1fd40b7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fcaffd640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fd40d2d40 0x7f1fd40b7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fcaffd640 1 -- 172.21.15.77:0/1414040269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd40b8630 con 0x7f1fd40d4a10 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fd37fe640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fd40d2d40 0x7f1fd40b7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fcaffd640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd40d4a10 0x7f1fd41513d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00e7e0 tx=0x7f1fbc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fd0ff9640 1 -- 172.21.15.77:0/1414040269 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc00f040 con 0x7f1fd40d4a10 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fd0ff9640 1 -- 172.21.15.77:0/1414040269 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fbc004030 con 0x7f1fd40d4a10 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fd0ff9640 1 -- 172.21.15.77:0/1414040269 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc0043e0 con 0x7f1fd40d4a10 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fda190640 1 -- 172.21.15.77:0/1414040269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd40b8910 con 0x7f1fd40d4a10 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.189+0000 7f1fda190640 1 -- 172.21.15.77:0/1414040269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd40b8ed0 con 0x7f1fd40d4a10 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.190+0000 7f1fda190640 1 -- 172.21.15.77:0/1414040269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fd4068fb0 con 0x7f1fd40d4a10 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.190+0000 7f1fd0ff9640 1 -- 172.21.15.77:0/1414040269 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1fbc00c6d0 con 0x7f1fd40d4a10 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.191+0000 7f1fd0ff9640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fb406be30 0x7f1fb406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.191+0000 7f1fd0ff9640 1 -- 172.21.15.77:0/1414040269 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f1fbc08acf0 con 0x7f1fd40d4a10 2024-04-25T01:24:10.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.191+0000 7f1fd37fe640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fb406be30 0x7f1fb406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.191+0000 7f1fd37fe640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fb406be30 0x7f1fb406e2f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009af0 tx=0x7f1fc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:10.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.195+0000 7f1fd0ff9640 1 -- 172.21.15.77:0/1414040269 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1fbc08f070 con 0x7f1fd40d4a10 2024-04-25T01:24:10.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.408+0000 7f1fda190640 1 -- 172.21.15.77:0/1414040269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1fd40d3c50 con 0x7f1fd40d4a10 2024-04-25T01:24:10.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.408+0000 7f1fd0ff9640 1 -- 172.21.15.77:0/1414040269 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5188 (secure 0 0 0) 0x7f1fbc017020 con 0x7f1fd40d4a10 2024-04-25T01:24:10.389 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:10.389 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5129,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1374960539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1374960539},{"type":"v1","addr":"172.21.15.121:6834","nonce":1374960539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5467,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3910406676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":3910406676},{"type":"v1","addr":"172.21.15.121:6835","nonce":3910406676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6120,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3088617511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3088617511},{"type":"v1","addr":"172.21.15.77:6837","nonce":3088617511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:23:28.170422+0000","modified":"2024-04-25T01:24:04.977155+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/2057169374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2057169374},{"type":"v1","addr":"172.21.15.77:6835","nonce":2057169374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6102,"qdb_cluster":[6102]},"id":3}]} 2024-04-25T01:24:10.390 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 36 2024-04-25T01:24:10.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.410+0000 7f1fda190640 1 -- 172.21.15.77:0/1414040269 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fb406be30 msgr2=0x7f1fb406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.410+0000 7f1fda190640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fb406be30 0x7f1fb406e2f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009af0 tx=0x7f1fc4008040 comp rx=0 tx=0).stop 2024-04-25T01:24:10.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.410+0000 7f1fda190640 1 -- 172.21.15.77:0/1414040269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd40d4a10 msgr2=0x7f1fd41513d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.410+0000 7f1fda190640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd40d4a10 0x7f1fd41513d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00e7e0 tx=0x7f1fbc00eca0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.410+0000 7f1fda190640 1 -- 172.21.15.77:0/1414040269 shutdown_connections 2024-04-25T01:24:10.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.410+0000 7f1fda190640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f1fb406be30 0x7f1fb406e2f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.410+0000 7f1fda190640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd40d4a10 0x7f1fd41513d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.410+0000 7f1fda190640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fd415e4c0 0x7f1fd40b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.410+0000 7f1fda190640 1 --2- 172.21.15.77:0/1414040269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fd40d2d40 0x7f1fd40b7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.410+0000 7f1fda190640 1 -- 172.21.15.77:0/1414040269 >> 172.21.15.77:0/1414040269 conn(0x7f1fd40d3ef0 msgr2=0x7f1fd40b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:10.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.410+0000 7f1fda190640 1 -- 172.21.15.77:0/1414040269 shutdown_connections 2024-04-25T01:24:10.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.411+0000 7f1fda190640 1 -- 172.21.15.77:0/1414040269 wait complete. 2024-04-25T01:24:10.402 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:24:10.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.518+0000 7fc78c121640 1 Processor -- start 2024-04-25T01:24:10.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.518+0000 7fc78c121640 1 -- start start 2024-04-25T01:24:10.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc78c121640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78415f9b0 0x7fc78415fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc78c121640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7840b4240 con 0x7fc78406d270 2024-04-25T01:24:10.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc78c121640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7840b39a0 con 0x7fc784145590 2024-04-25T01:24:10.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc78c121640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc784160830 con 0x7fc78415f9b0 2024-04-25T01:24:10.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc78a6be640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc78406d270 0x7fc78406d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49170/0 (socket says 172.21.15.77:49170) 2024-04-25T01:24:10.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc78a6be640 1 -- 172.21.15.77:0/3316653200 learned_addr learned my addr 172.21.15.77:0/3316653200 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:10.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc7896bc640 1 --2- 172.21.15.77:0/3316653200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78415f9b0 0x7fc78415fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc788ebb640 1 -- 172.21.15.77:0/3316653200 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1593178884 0 0) 0x7fc7840b4240 con 0x7fc78406d270 2024-04-25T01:24:10.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc788ebb640 1 -- 172.21.15.77:0/3316653200 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc770003520 con 0x7fc78406d270 2024-04-25T01:24:10.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc788ebb640 1 -- 172.21.15.77:0/3316653200 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1992806653 0 0) 0x7fc7840b39a0 con 0x7fc784145590 2024-04-25T01:24:10.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc788ebb640 1 -- 172.21.15.77:0/3316653200 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7840b4240 con 0x7fc784145590 2024-04-25T01:24:10.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc7896bc640 1 -- 172.21.15.77:0/3316653200 >> v1:172.21.15.121:6789/0 conn(0x7fc784145590 legacy=0x7fc7840b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc788ebb640 1 -- 172.21.15.77:0/3316653200 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2743261457 0 0) 0x7fc770003520 con 0x7fc78406d270 2024-04-25T01:24:10.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc7896bc640 1 -- 172.21.15.77:0/3316653200 >> v1:172.21.15.77:6789/0 conn(0x7fc78406d270 legacy=0x7fc78406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.519+0000 7fc7896bc640 1 -- 172.21.15.77:0/3316653200 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7840b3c80 con 0x7fc78415f9b0 2024-04-25T01:24:10.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.520+0000 7fc7896bc640 1 --2- 172.21.15.77:0/3316653200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78415f9b0 0x7fc78415fe10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc7740097d0 tx=0x7fc77402a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b02d349f8f019682 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.520+0000 7fc788ebb640 1 -- 172.21.15.77:0/3316653200 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774037070 con 0x7fc78415f9b0 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.520+0000 7fc788ebb640 1 -- 172.21.15.77:0/3316653200 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc77402ac30 con 0x7fc78415f9b0 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.520+0000 7fc788ebb640 1 -- 172.21.15.77:0/3316653200 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c0019f0 con 0x7fc78415f9b0 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.521+0000 7fc78c121640 1 -- 172.21.15.77:0/3316653200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78415f9b0 msgr2=0x7fc78415fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.521+0000 7fc78c121640 1 --2- 172.21.15.77:0/3316653200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78415f9b0 0x7fc78415fe10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc7740097d0 tx=0x7fc77402a1a0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.521+0000 7fc78c121640 1 -- 172.21.15.77:0/3316653200 shutdown_connections 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.521+0000 7fc78c121640 1 --2- 172.21.15.77:0/3316653200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78415f9b0 0x7fc78415fe10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.521+0000 7fc78c121640 1 -- 172.21.15.77:0/3316653200 >> 172.21.15.77:0/3316653200 conn(0x7fc7840b1820 msgr2=0x7fc7840b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.521+0000 7fc78c121640 1 -- 172.21.15.77:0/3316653200 shutdown_connections 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.521+0000 7fc78c121640 1 -- 172.21.15.77:0/3316653200 wait complete. 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.522+0000 7fc78c121640 1 Processor -- start 2024-04-25T01:24:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.522+0000 7fc78c121640 1 -- start start 2024-04-25T01:24:10.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78c121640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78406d270 0x7fc7840aa7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78c121640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc784145590 0x7fc7841402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78c121640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc78415f9b0 0x7fc7840aaec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78c121640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc78406cde0 con 0x7fc78415f9b0 2024-04-25T01:24:10.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78c121640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc784145970 con 0x7fc78406d270 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78c121640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc7840b3530 con 0x7fc784145590 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78a6be640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc78415f9b0 0x7fc7840aaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78a6be640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc78415f9b0 0x7fc7840aaec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48562/0 (socket says 172.21.15.77:48562) 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78a6be640 1 -- 172.21.15.77:0/3662539432 learned_addr learned my addr 172.21.15.77:0/3662539432 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc7896bc640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc784145590 0x7fc7841402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc789ebd640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78406d270 0x7fc7840aa7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78a6be640 1 -- 172.21.15.77:0/3662539432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc784145590 msgr2=0x7fc7841402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78a6be640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc784145590 0x7fc7841402c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78a6be640 1 -- 172.21.15.77:0/3662539432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78406d270 msgr2=0x7fc7840aa7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78a6be640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78406d270 0x7fc7840aa7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78a6be640 1 -- 172.21.15.77:0/3662539432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7840ab660 con 0x7fc78415f9b0 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78a6be640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc78415f9b0 0x7fc7840aaec0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc77c001350 tx=0x7fc77c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:10.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc77a7fc640 1 -- 172.21.15.77:0/3662539432 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c017070 con 0x7fc78415f9b0 2024-04-25T01:24:10.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc77a7fc640 1 -- 172.21.15.77:0/3662539432 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc77c012400 con 0x7fc78415f9b0 2024-04-25T01:24:10.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78c121640 1 -- 172.21.15.77:0/3662539432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7840ab940 con 0x7fc78415f9b0 2024-04-25T01:24:10.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc77a7fc640 1 -- 172.21.15.77:0/3662539432 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c012830 con 0x7fc78415f9b0 2024-04-25T01:24:10.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.523+0000 7fc78c121640 1 -- 172.21.15.77:0/3662539432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7840abf00 con 0x7fc78415f9b0 2024-04-25T01:24:10.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.524+0000 7fc78c121640 1 -- 172.21.15.77:0/3662539432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc784068fb0 con 0x7fc78415f9b0 2024-04-25T01:24:10.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.525+0000 7fc77a7fc640 1 -- 172.21.15.77:0/3662539432 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc77c01a070 con 0x7fc78415f9b0 2024-04-25T01:24:10.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.525+0000 7fc77a7fc640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc76806be30 0x7fc76806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.525+0000 7fc77a7fc640 1 -- 172.21.15.77:0/3662539432 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7fc77c08aea0 con 0x7fc78415f9b0 2024-04-25T01:24:10.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.525+0000 7fc789ebd640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc76806be30 0x7fc76806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.526+0000 7fc789ebd640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc76806be30 0x7fc76806e2f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc780000f60 tx=0x7fc780009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:10.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.529+0000 7fc77a7fc640 1 -- 172.21.15.77:0/3662539432 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc77c059050 con 0x7fc78415f9b0 2024-04-25T01:24:10.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.704+0000 7fc78c121640 1 -- 172.21.15.77:0/3662539432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc78406ea50 con 0x7fc78415f9b0 2024-04-25T01:24:10.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.705+0000 7fc77a7fc640 1 -- 172.21.15.77:0/3662539432 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+17203 (secure 0 0 0) 0x7fc77c05d350 con 0x7fc78415f9b0 2024-04-25T01:24:10.686 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:10.686 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":55,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:24:01.947173+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:23:26.392274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T01:23:27.146093+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6836/3189776231":"2024-04-26T01:24:01.947151+0000","172.21.15.77:6834/3189776231":"2024-04-26T01:24:01.947151+0000","172.21.15.121:6834/3834135103":"2024-04-26T01:24:00.948815+0000","172.21.15.121:6832/3834135103":"2024-04-26T01:24:00.948815+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:24:10.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 -- 172.21.15.77:0/3662539432 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc76806be30 msgr2=0x7fc76806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc76806be30 0x7fc76806e2f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc780000f60 tx=0x7fc780009040 comp rx=0 tx=0).stop 2024-04-25T01:24:10.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 -- 172.21.15.77:0/3662539432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc78415f9b0 msgr2=0x7fc7840aaec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc78415f9b0 0x7fc7840aaec0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc77c001350 tx=0x7fc77c00cf90 comp rx=0 tx=0).stop 2024-04-25T01:24:10.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 -- 172.21.15.77:0/3662539432 shutdown_connections 2024-04-25T01:24:10.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc76806be30 0x7fc76806e2f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc78415f9b0 0x7fc7840aaec0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc784145590 0x7fc7841402c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 --2- 172.21.15.77:0/3662539432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc78406d270 0x7fc7840aa7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 -- 172.21.15.77:0/3662539432 >> 172.21.15.77:0/3662539432 conn(0x7fc7840b1820 msgr2=0x7fc7840afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:10.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 -- 172.21.15.77:0/3662539432 shutdown_connections 2024-04-25T01:24:10.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.707+0000 7fc78c121640 1 -- 172.21.15.77:0/3662539432 wait complete. 2024-04-25T01:24:10.699 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T01:24:10.699 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:10.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.815+0000 7f094f377640 1 Processor -- start 2024-04-25T01:24:10.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.815+0000 7f094f377640 1 -- start start 2024-04-25T01:24:10.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094f377640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f094815f9b0 0x7f094815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094f377640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f094806d270 0x7f094806d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094f377640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09480b4240 con 0x7f0948145590 2024-04-25T01:24:10.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094f377640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f0948160830 con 0x7f094815f9b0 2024-04-25T01:24:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094f377640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0948160350 con 0x7f094806d270 2024-04-25T01:24:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094c912640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f094815f9b0 0x7f094815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094c912640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f094815f9b0 0x7f094815fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48564/0 (socket says 172.21.15.77:48564) 2024-04-25T01:24:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094c912640 1 -- 172.21.15.77:0/1326855148 learned_addr learned my addr 172.21.15.77:0/1326855148 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094d914640 1 --2- 172.21.15.77:0/1326855148 >> v2:172.21.15.121:3300/0 conn(0x7f094806d270 0x7f094806d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094c912640 1 -- 172.21.15.77:0/1326855148 >> v2:172.21.15.121:3300/0 conn(0x7f094806d270 msgr2=0x7f094806d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094c912640 1 --2- 172.21.15.77:0/1326855148 >> v2:172.21.15.121:3300/0 conn(0x7f094806d270 0x7f094806d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094c912640 1 -- 172.21.15.77:0/1326855148 >> v1:172.21.15.121:6789/0 conn(0x7f0948145590 legacy=0x7f09480b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.816+0000 7f094c912640 1 -- 172.21.15.77:0/1326855148 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09480b39a0 con 0x7f094815f9b0 2024-04-25T01:24:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.817+0000 7f094d914640 1 --2- 172.21.15.77:0/1326855148 >> v2:172.21.15.121:3300/0 conn(0x7f094806d270 0x7f094806d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.817+0000 7f094c912640 1 --2- 172.21.15.77:0/1326855148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f094815f9b0 0x7f094815fe10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0948068b60 tx=0x7f093802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da921b7af24f533e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.817+0000 7f093ffff640 1 -- 172.21.15.77:0/1326855148 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938037070 con 0x7f094815f9b0 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.817+0000 7f093ffff640 1 -- 172.21.15.77:0/1326855148 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0938032440 con 0x7f094815f9b0 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.817+0000 7f093ffff640 1 -- 172.21.15.77:0/1326855148 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938032810 con 0x7f094815f9b0 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.818+0000 7f094f377640 1 -- 172.21.15.77:0/1326855148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f094815f9b0 msgr2=0x7f094815fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.818+0000 7f094f377640 1 --2- 172.21.15.77:0/1326855148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f094815f9b0 0x7f094815fe10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0948068b60 tx=0x7f093802a270 comp rx=0 tx=0).stop 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.818+0000 7f094f377640 1 -- 172.21.15.77:0/1326855148 shutdown_connections 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.818+0000 7f094f377640 1 --2- 172.21.15.77:0/1326855148 >> v2:172.21.15.121:3300/0 conn(0x7f094806d270 0x7f094806d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.818+0000 7f094f377640 1 --2- 172.21.15.77:0/1326855148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f094815f9b0 0x7f094815fe10 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.818+0000 7f094f377640 1 -- 172.21.15.77:0/1326855148 >> 172.21.15.77:0/1326855148 conn(0x7f09480b1820 msgr2=0x7f09480b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.818+0000 7f094f377640 1 -- 172.21.15.77:0/1326855148 shutdown_connections 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.818+0000 7f094f377640 1 -- 172.21.15.77:0/1326855148 wait complete. 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.819+0000 7f094f377640 1 Processor -- start 2024-04-25T01:24:10.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.819+0000 7f094f377640 1 -- start start 2024-04-25T01:24:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094f377640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806d270 0x7f0948153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094f377640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0948145590 0x7f0948072600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094f377640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f094815f9b0 0x7f094813b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094f377640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f09480b3f60 con 0x7f094815f9b0 2024-04-25T01:24:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094f377640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f094806cde0 con 0x7f0948145590 2024-04-25T01:24:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094f377640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0948145970 con 0x7f094806d270 2024-04-25T01:24:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094d914640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f094815f9b0 0x7f094813b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094d914640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f094815f9b0 0x7f094813b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48576/0 (socket says 172.21.15.77:48576) 2024-04-25T01:24:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094d914640 1 -- 172.21.15.77:0/1836255663 learned_addr learned my addr 172.21.15.77:0/1836255663 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094c912640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0948145590 0x7f0948072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094d914640 1 -- 172.21.15.77:0/1836255663 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806d270 msgr2=0x7f0948153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094d113640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806d270 0x7f0948153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094d914640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806d270 0x7f0948153520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094d914640 1 -- 172.21.15.77:0/1836255663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0948145590 msgr2=0x7f0948072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094d914640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0948145590 0x7f0948072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094d914640 1 -- 172.21.15.77:0/1836255663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094815c9d0 con 0x7f094815f9b0 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094c912640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0948145590 0x7f0948072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094d113640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806d270 0x7f0948153520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f094d914640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f094815f9b0 0x7f094813b930 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f094000e7e0 tx=0x7f094000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.820+0000 7f093dffb640 1 -- 172.21.15.77:0/1836255663 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094000f040 con 0x7f094815f9b0 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.821+0000 7f093dffb640 1 -- 172.21.15.77:0/1836255663 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0940004030 con 0x7f094815f9b0 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.821+0000 7f094f377640 1 -- 172.21.15.77:0/1836255663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f094815cf60 con 0x7f094815f9b0 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.821+0000 7f093dffb640 1 -- 172.21.15.77:0/1836255663 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09400043e0 con 0x7f094815f9b0 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.821+0000 7f094f377640 1 -- 172.21.15.77:0/1836255663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f094815d520 con 0x7f094815f9b0 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.822+0000 7f094f377640 1 -- 172.21.15.77:0/1836255663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0948068fb0 con 0x7f094815f9b0 2024-04-25T01:24:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.822+0000 7f093dffb640 1 -- 172.21.15.77:0/1836255663 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f094000c670 con 0x7f094815f9b0 2024-04-25T01:24:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.823+0000 7f093dffb640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f092c06be30 0x7f092c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.823+0000 7f093dffb640 1 -- 172.21.15.77:0/1836255663 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f094008acc0 con 0x7f094815f9b0 2024-04-25T01:24:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.823+0000 7f094d113640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f092c06be30 0x7f092c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.823+0000 7f094d113640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f092c06be30 0x7f092c06e2f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0944001d70 tx=0x7f0944009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:10.827+0000 7f093dffb640 1 -- 172.21.15.77:0/1836255663 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0940017020 con 0x7f094815f9b0 2024-04-25T01:24:11.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.038+0000 7f094f377640 1 -- 172.21.15.77:0/1836255663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f094806ea50 con 0x7f094815f9b0 2024-04-25T01:24:11.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.039+0000 7f093dffb640 1 -- 172.21.15.77:0/1836255663 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5188 (secure 0 0 0) 0x7f09400594c0 con 0x7f094815f9b0 2024-04-25T01:24:11.020 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:11.020 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5129,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1374960539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1374960539},{"type":"v1","addr":"172.21.15.121:6834","nonce":1374960539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5467,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3910406676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":3910406676},{"type":"v1","addr":"172.21.15.121:6835","nonce":3910406676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6120,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3088617511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3088617511},{"type":"v1","addr":"172.21.15.77:6837","nonce":3088617511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:23:28.170422+0000","modified":"2024-04-25T01:24:04.977155+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/2057169374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2057169374},{"type":"v1","addr":"172.21.15.77:6835","nonce":2057169374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6102,"qdb_cluster":[6102]},"id":3}]} 2024-04-25T01:24:11.020 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 36 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 -- 172.21.15.77:0/1836255663 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f092c06be30 msgr2=0x7f092c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f092c06be30 0x7f092c06e2f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0944001d70 tx=0x7f0944009040 comp rx=0 tx=0).stop 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 -- 172.21.15.77:0/1836255663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f094815f9b0 msgr2=0x7f094813b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f094815f9b0 0x7f094813b930 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f094000e7e0 tx=0x7f094000eca0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 -- 172.21.15.77:0/1836255663 shutdown_connections 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f092c06be30 0x7f092c06e2f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f094815f9b0 0x7f094813b930 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0948145590 0x7f0948072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 --2- 172.21.15.77:0/1836255663 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806d270 0x7f0948153520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 -- 172.21.15.77:0/1836255663 >> 172.21.15.77:0/1836255663 conn(0x7f09480b1820 msgr2=0x7f0948141700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 -- 172.21.15.77:0/1836255663 shutdown_connections 2024-04-25T01:24:11.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.041+0000 7f094f377640 1 -- 172.21.15.77:0/1836255663 wait complete. 2024-04-25T01:24:11.033 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:11.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.147+0000 7f7dd604d640 1 Processor -- start 2024-04-25T01:24:11.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.148+0000 7f7dd604d640 1 -- start start 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.148+0000 7f7dd604d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd015e4c0 0x7f7dd00d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.148+0000 7f7dd604d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dd00d1e90 con 0x7f7dd00d2d40 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.148+0000 7f7dd604d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dd00b54a0 con 0x7f7dd00d4a10 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.148+0000 7f7dd604d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd00d2130 con 0x7f7dd015e4c0 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.148+0000 7f7dcffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7dd00d2d40 0x7f7dd00d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49182/0 (socket says 172.21.15.77:49182) 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.148+0000 7f7dcffff640 1 -- 172.21.15.77:0/680977607 learned_addr learned my addr 172.21.15.77:0/680977607 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2673623906 0 0) 0x7f7dd00d1e90 con 0x7f7dd00d2d40 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dcf7fe640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd015e4c0 0x7f7dd00d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7dbc003520 con 0x7f7dd00d2d40 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4145245614 0 0) 0x7f7dbc003520 con 0x7f7dd00d2d40 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd015e4c0 msgr2=0x7f7dd00d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd015e4c0 0x7f7dd00d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 >> v1:172.21.15.121:6789/0 conn(0x7f7dd00d4a10 legacy=0x7f7dd00b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd00b5780 con 0x7f7dd00d2d40 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f7dc80019f0 con 0x7f7dd00d2d40 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 >> v1:172.21.15.77:6789/0 conn(0x7f7dd00d2d40 legacy=0x7f7dd00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc0039e0 0x7f7dbc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dbc004390 0x7f7dbc004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd015e4c0 0x7f7dbc005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd00d2130 con 0x7f7dbc0039e0 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7dbc00ba70 con 0x7f7dbc004390 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7dbc00bc70 con 0x7f7dd015e4c0 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dcf7fe640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc0039e0 0x7f7dbc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dcffff640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd015e4c0 0x7f7dbc005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dc6ffd640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dbc004390 0x7f7dbc004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dcf7fe640 1 -- 172.21.15.77:0/680977607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd015e4c0 msgr2=0x7f7dbc005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dcf7fe640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd015e4c0 0x7f7dbc005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dcf7fe640 1 -- 172.21.15.77:0/680977607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dbc004390 msgr2=0x7f7dbc004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:11.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dcf7fe640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dbc004390 0x7f7dbc004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.149+0000 7f7dcf7fe640 1 -- 172.21.15.77:0/680977607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dbc003520 con 0x7f7dbc0039e0 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.150+0000 7f7dcf7fe640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc0039e0 0x7f7dbc003dc0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0009ac0 tx=0x7f7dbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01c88b1e235093b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.150+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0036070 con 0x7f7dbc0039e0 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.150+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc0031400 con 0x7f7dbc0039e0 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.150+0000 7f7dceffd640 1 -- 172.21.15.77:0/680977607 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0009690 con 0x7f7dbc0039e0 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.151+0000 7f7dd604d640 1 -- 172.21.15.77:0/680977607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc0039e0 msgr2=0x7f7dbc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.151+0000 7f7dd604d640 1 --2- 172.21.15.77:0/680977607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc0039e0 0x7f7dbc003dc0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0009ac0 tx=0x7f7dbc005830 comp rx=0 tx=0).stop 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.151+0000 7f7dcffff640 1 -- 172.21.15.77:0/680977607 reap_dead start 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.151+0000 7f7dd604d640 1 -- 172.21.15.77:0/680977607 shutdown_connections 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.151+0000 7f7dd604d640 1 -- 172.21.15.77:0/680977607 >> 172.21.15.77:0/680977607 conn(0x7f7dd00d3ef0 msgr2=0x7f7dd00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.151+0000 7f7dd604d640 1 -- 172.21.15.77:0/680977607 shutdown_connections 2024-04-25T01:24:11.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.151+0000 7f7dd604d640 1 -- 172.21.15.77:0/680977607 wait complete. 2024-04-25T01:24:11.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.152+0000 7f7dd604d640 1 Processor -- start 2024-04-25T01:24:11.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.152+0000 7f7dd604d640 1 -- start start 2024-04-25T01:24:11.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dd604d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dbc0039e0 0x7f7dd00cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dd604d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd0152570 0x7f7dd00c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dd604d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd00c9f70 0x7f7dd0152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dd604d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd015ea20 con 0x7f7dd00c9f70 2024-04-25T01:24:11.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dd604d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd015e8a0 con 0x7f7dbc0039e0 2024-04-25T01:24:11.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dd604d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd00b45f0 con 0x7f7dd0152570 2024-04-25T01:24:11.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dcffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd00c9f70 0x7f7dd0152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dcffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd00c9f70 0x7f7dd0152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48592/0 (socket says 172.21.15.77:48592) 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dcffff640 1 -- 172.21.15.77:0/763227286 learned_addr learned my addr 172.21.15.77:0/763227286 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dcf7fe640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dbc0039e0 0x7f7dd00cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dc6ffd640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd0152570 0x7f7dd00c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dcffff640 1 -- 172.21.15.77:0/763227286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd0152570 msgr2=0x7f7dd00c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dcffff640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd0152570 0x7f7dd00c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dcffff640 1 -- 172.21.15.77:0/763227286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dbc0039e0 msgr2=0x7f7dd00cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dcffff640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dbc0039e0 0x7f7dd00cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dcffff640 1 -- 172.21.15.77:0/763227286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd00cc5f0 con 0x7f7dd00c9f70 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dcf7fe640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dbc0039e0 0x7f7dd00cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.153+0000 7f7dcffff640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd00c9f70 0x7f7dd0152dc0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7dd006b940 tx=0x7f7dc800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.154+0000 7f7dccff9640 1 -- 172.21.15.77:0/763227286 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc800ec40 con 0x7f7dd00c9f70 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.154+0000 7f7dccff9640 1 -- 172.21.15.77:0/763227286 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc800ede0 con 0x7f7dd00c9f70 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.154+0000 7f7dd604d640 1 -- 172.21.15.77:0/763227286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd00cc8d0 con 0x7f7dd00c9f70 2024-04-25T01:24:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.154+0000 7f7dccff9640 1 -- 172.21.15.77:0/763227286 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8014670 con 0x7f7dd00c9f70 2024-04-25T01:24:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.154+0000 7f7dd604d640 1 -- 172.21.15.77:0/763227286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd00cce90 con 0x7f7dd00c9f70 2024-04-25T01:24:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.155+0000 7f7dd604d640 1 -- 172.21.15.77:0/763227286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dd00ca3d0 con 0x7f7dd00c9f70 2024-04-25T01:24:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.155+0000 7f7dccff9640 1 -- 172.21.15.77:0/763227286 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7dc8014810 con 0x7f7dd00c9f70 2024-04-25T01:24:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.155+0000 7f7dccff9640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7db006be30 0x7f7db006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.156+0000 7f7dccff9640 1 -- 172.21.15.77:0/763227286 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f7dc808aec0 con 0x7f7dd00c9f70 2024-04-25T01:24:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.156+0000 7f7dcf7fe640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7db006be30 0x7f7db006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.156+0000 7f7dcf7fe640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7db006be30 0x7f7db006e2f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0033000 tx=0x7f7dc0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:11.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.160+0000 7f7dccff9640 1 -- 172.21.15.77:0/763227286 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7dc8059630 con 0x7f7dd00c9f70 2024-04-25T01:24:11.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.369+0000 7f7dd604d640 1 -- 172.21.15.77:0/763227286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7dd0068fb0 con 0x7f7dd00c9f70 2024-04-25T01:24:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.370+0000 7f7dccff9640 1 -- 172.21.15.77:0/763227286 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5188 (secure 0 0 0) 0x7f7dc8058f30 con 0x7f7dd00c9f70 2024-04-25T01:24:11.351 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:11.351 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5129,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1374960539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1374960539},{"type":"v1","addr":"172.21.15.121:6834","nonce":1374960539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5467,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3910406676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":3910406676},{"type":"v1","addr":"172.21.15.121:6835","nonce":3910406676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6120,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3088617511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3088617511},{"type":"v1","addr":"172.21.15.77:6837","nonce":3088617511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:23:28.170422+0000","modified":"2024-04-25T01:24:04.977155+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/2057169374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2057169374},{"type":"v1","addr":"172.21.15.77:6835","nonce":2057169374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6102,"qdb_cluster":[6102]},"id":3}]} 2024-04-25T01:24:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 36 2024-04-25T01:24:11.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 -- 172.21.15.77:0/763227286 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7db006be30 msgr2=0x7f7db006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7db006be30 0x7f7db006e2f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0033000 tx=0x7f7dc0008030 comp rx=0 tx=0).stop 2024-04-25T01:24:11.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 -- 172.21.15.77:0/763227286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd00c9f70 msgr2=0x7f7dd0152dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd00c9f70 0x7f7dd0152dc0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7dd006b940 tx=0x7f7dc800e530 comp rx=0 tx=0).stop 2024-04-25T01:24:11.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 -- 172.21.15.77:0/763227286 shutdown_connections 2024-04-25T01:24:11.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f7db006be30 0x7f7db006e2f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd00c9f70 0x7f7dd0152dc0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7dd0152570 0x7f7dd00c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 --2- 172.21.15.77:0/763227286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dbc0039e0 0x7f7dd00cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 -- 172.21.15.77:0/763227286 >> 172.21.15.77:0/763227286 conn(0x7f7dd00d3ef0 msgr2=0x7f7dd015d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:11.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 -- 172.21.15.77:0/763227286 shutdown_connections 2024-04-25T01:24:11.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.372+0000 7f7dd604d640 1 -- 172.21.15.77:0/763227286 wait complete. 2024-04-25T01:24:11.364 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:24:11.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.480+0000 7f28c2589640 1 Processor -- start 2024-04-25T01:24:11.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.480+0000 7f28c2589640 1 -- start start 2024-04-25T01:24:11.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.480+0000 7f28c2589640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc0cfaf0 0x7f28bc0d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.480+0000 7f28c2589640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28bc0d1e90 con 0x7f28bc15e4c0 2024-04-25T01:24:11.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.480+0000 7f28c2589640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28bc0b54a0 con 0x7f28bc0d4a10 2024-04-25T01:24:11.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.480+0000 7f28c2589640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f28bc0d2130 con 0x7f28bc0cfaf0 2024-04-25T01:24:11.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28bbfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f28bc15e4c0 0x7f28bc0d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49186/0 (socket says 172.21.15.77:49186) 2024-04-25T01:24:11.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28bbfff640 1 -- 172.21.15.77:0/4158333459 learned_addr learned my addr 172.21.15.77:0/4158333459 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:11.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28c0b26640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc0cfaf0 0x7f28bc0d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1024578531 0 0) 0x7f28bc0d1e90 con 0x7f28bc15e4c0 2024-04-25T01:24:11.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28a4003520 con 0x7f28bc15e4c0 2024-04-25T01:24:11.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1952614707 0 0) 0x7f28bc0b54a0 con 0x7f28bc0d4a10 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28bc0d1e90 con 0x7f28bc0d4a10 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 61126037 0 0) 0x7f28a4003520 con 0x7f28bc15e4c0 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc0cfaf0 msgr2=0x7f28bc0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc0cfaf0 0x7f28bc0d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 >> v1:172.21.15.121:6789/0 conn(0x7f28bc0d4a10 legacy=0x7f28bc0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28bc0b5780 con 0x7f28bc15e4c0 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f28b0001670 con 0x7f28bc15e4c0 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28c0b26640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc0cfaf0 0x7f28bc0d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 >> v1:172.21.15.77:6789/0 conn(0x7f28bc15e4c0 legacy=0x7f28bc0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a4003b70 0x7f28a4003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc0cfaf0 0x7f28a4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a4004d40 0x7f28a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f28bc0d2130 con 0x7f28a4004d40 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f28a400baa0 con 0x7f28a4003b70 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f28a400bca0 con 0x7f28bc0cfaf0 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28c0b26640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a4004d40 0x7f28a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28bbfff640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a4003b70 0x7f28a4003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.481+0000 7f28bb7fe640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc0cfaf0 0x7f28a4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.482+0000 7f28c0b26640 1 -- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc0cfaf0 msgr2=0x7f28a4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.482+0000 7f28c0b26640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc0cfaf0 0x7f28a4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.482+0000 7f28c0b26640 1 -- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a4003b70 msgr2=0x7f28a4003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.482+0000 7f28c0b26640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a4003b70 0x7f28a4003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.482+0000 7f28c0b26640 1 -- 172.21.15.77:0/4158333459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a4003520 con 0x7f28a4004d40 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.482+0000 7f28c0b26640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a4004d40 0x7f28a40051c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f28b400e350 tx=0x7f28a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571a9d28cc1ec8cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.482+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4033070 con 0x7f28a4004d40 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.482+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b402f660 con 0x7f28a4004d40 2024-04-25T01:24:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.482+0000 7f28baffd640 1 -- 172.21.15.77:0/4158333459 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4009de0 con 0x7f28a4004d40 2024-04-25T01:24:11.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.483+0000 7f28c2589640 1 -- 172.21.15.77:0/4158333459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a4004d40 msgr2=0x7f28a40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.483+0000 7f28c2589640 1 --2- 172.21.15.77:0/4158333459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28a4004d40 0x7f28a40051c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f28b400e350 tx=0x7f28a4005830 comp rx=0 tx=0).stop 2024-04-25T01:24:11.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.483+0000 7f28c0b26640 1 -- 172.21.15.77:0/4158333459 reap_dead start 2024-04-25T01:24:11.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.483+0000 7f28c2589640 1 -- 172.21.15.77:0/4158333459 shutdown_connections 2024-04-25T01:24:11.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.483+0000 7f28c2589640 1 -- 172.21.15.77:0/4158333459 >> 172.21.15.77:0/4158333459 conn(0x7f28bc0d3ef0 msgr2=0x7f28bc0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:11.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.483+0000 7f28c2589640 1 -- 172.21.15.77:0/4158333459 shutdown_connections 2024-04-25T01:24:11.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.483+0000 7f28c2589640 1 -- 172.21.15.77:0/4158333459 wait complete. 2024-04-25T01:24:11.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.484+0000 7f28c2589640 1 Processor -- start 2024-04-25T01:24:11.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.484+0000 7f28c2589640 1 -- start start 2024-04-25T01:24:11.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28c2589640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc14d500 0x7f28bc159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28c2589640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28bc14d8e0 0x7f28bc14dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28c2589640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc157d90 0x7f28bc158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28c2589640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f28bc0b30a0 con 0x7f28bc14d8e0 2024-04-25T01:24:11.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28c2589640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f28bc0b2f20 con 0x7f28bc14d500 2024-04-25T01:24:11.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28c2589640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f28bc15e8a0 con 0x7f28bc157d90 2024-04-25T01:24:11.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28bc14d8e0 0x7f28bc14dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28bc14d8e0 0x7f28bc14dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48614/0 (socket says 172.21.15.77:48614) 2024-04-25T01:24:11.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28bb7fe640 1 -- 172.21.15.77:0/1654471758 learned_addr learned my addr 172.21.15.77:0/1654471758 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:11.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28bbfff640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc14d500 0x7f28bc159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28c0b26640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc157d90 0x7f28bc158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28bb7fe640 1 -- 172.21.15.77:0/1654471758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc157d90 msgr2=0x7f28bc158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28bb7fe640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc157d90 0x7f28bc158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28bb7fe640 1 -- 172.21.15.77:0/1654471758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc14d500 msgr2=0x7f28bc159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28bb7fe640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc14d500 0x7f28bc159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28bb7fe640 1 -- 172.21.15.77:0/1654471758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28bc15a300 con 0x7f28bc14d8e0 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.485+0000 7f28bbfff640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc14d500 0x7f28bc159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.486+0000 7f28bb7fe640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28bc14d8e0 0x7f28bc14dd60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f28ac00f7a0 tx=0x7f28ac00fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.486+0000 7f28b8ff9640 1 -- 172.21.15.77:0/1654471758 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac00fe50 con 0x7f28bc14d8e0 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.486+0000 7f28b8ff9640 1 -- 172.21.15.77:0/1654471758 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28ac018070 con 0x7f28bc14d8e0 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.486+0000 7f28c2589640 1 -- 172.21.15.77:0/1654471758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28bc15b100 con 0x7f28bc14d8e0 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.486+0000 7f28b8ff9640 1 -- 172.21.15.77:0/1654471758 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac00d730 con 0x7f28bc14d8e0 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.486+0000 7f28c2589640 1 -- 172.21.15.77:0/1654471758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28bc15a860 con 0x7f28bc14d8e0 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.487+0000 7f28c2589640 1 -- 172.21.15.77:0/1654471758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28bc068fb0 con 0x7f28bc14d8e0 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.487+0000 7f28b8ff9640 1 -- 172.21.15.77:0/1654471758 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f28ac00d8d0 con 0x7f28bc14d8e0 2024-04-25T01:24:11.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.488+0000 7f28b8ff9640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28a806be60 0x7f28a806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.488+0000 7f28b8ff9640 1 -- 172.21.15.77:0/1654471758 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f28ac08af20 con 0x7f28bc14d8e0 2024-04-25T01:24:11.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.488+0000 7f28bbfff640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28a806be60 0x7f28a806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.488+0000 7f28bbfff640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28a806be60 0x7f28a806e320 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f28b0004840 tx=0x7f28b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:11.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.492+0000 7f28b8ff9640 1 -- 172.21.15.77:0/1654471758 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f28bc15a300 con 0x7f28bc14d8e0 2024-04-25T01:24:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.664+0000 7f28c2589640 1 -- 172.21.15.77:0/1654471758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28bc0b0c10 con 0x7f28bc14d8e0 2024-04-25T01:24:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.665+0000 7f28b8ff9640 1 -- 172.21.15.77:0/1654471758 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+17203 (secure 0 0 0) 0x7f28ac01a310 con 0x7f28bc14d8e0 2024-04-25T01:24:11.646 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:11.646 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":55,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:24:01.947173+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:23:26.392274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T01:23:27.146093+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6836/3189776231":"2024-04-26T01:24:01.947151+0000","172.21.15.77:6834/3189776231":"2024-04-26T01:24:01.947151+0000","172.21.15.121:6834/3834135103":"2024-04-26T01:24:00.948815+0000","172.21.15.121:6832/3834135103":"2024-04-26T01:24:00.948815+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 -- 172.21.15.77:0/1654471758 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28a806be60 msgr2=0x7f28a806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28a806be60 0x7f28a806e320 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f28b0004840 tx=0x7f28b0009040 comp rx=0 tx=0).stop 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 -- 172.21.15.77:0/1654471758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28bc14d8e0 msgr2=0x7f28bc14dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28bc14d8e0 0x7f28bc14dd60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f28ac00f7a0 tx=0x7f28ac00fc60 comp rx=0 tx=0).stop 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 -- 172.21.15.77:0/1654471758 shutdown_connections 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f28a806be60 0x7f28a806e320 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28bc14d8e0 0x7f28bc14dd60 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28bc157d90 0x7f28bc158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 --2- 172.21.15.77:0/1654471758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc14d500 0x7f28bc159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 -- 172.21.15.77:0/1654471758 >> 172.21.15.77:0/1654471758 conn(0x7f28bc0d3ef0 msgr2=0x7f28bc0b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 -- 172.21.15.77:0/1654471758 shutdown_connections 2024-04-25T01:24:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.667+0000 7f28c2589640 1 -- 172.21.15.77:0/1654471758 wait complete. 2024-04-25T01:24:11.659 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T01:24:11.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.776+0000 7f137c77d640 1 Processor -- start 2024-04-25T01:24:11.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.776+0000 7f137c77d640 1 -- start start 2024-04-25T01:24:11.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f137c77d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f137415b5f0 0x7f13740711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f137c77d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1374071730 0x7f137406e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f137c77d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1374154ea0 con 0x7f1374154ac0 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f137c77d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f137406d790 con 0x7f1374071730 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f137c77d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f137406da70 con 0x7f137415b5f0 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f137a519640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1374154ac0 0x7f13740cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49196/0 (socket says 172.21.15.77:49196) 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f137a519640 1 -- 172.21.15.77:0/4235748801 learned_addr learned my addr 172.21.15.77:0/4235748801 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2934119931 0 0) 0x7f1374154ea0 con 0x7f1374154ac0 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f137ad1a640 1 --2- 172.21.15.77:0/4235748801 >> v2:172.21.15.121:3300/0 conn(0x7f1374071730 0x7f137406e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1360003520 con 0x7f1374154ac0 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379d18640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f137415b5f0 0x7f13740711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 941899295 0 0) 0x7f1360003520 con 0x7f1374154ac0 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f137415b5f0 msgr2=0x7f13740711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f137415b5f0 0x7f13740711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 >> v2:172.21.15.121:3300/0 conn(0x7f1374071730 msgr2=0x7f137406e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 --2- 172.21.15.77:0/4235748801 >> v2:172.21.15.121:3300/0 conn(0x7f1374071730 0x7f137406e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137406dd50 con 0x7f1374154ac0 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f1370001670 con 0x7f1374154ac0 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379d18640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f137415b5f0 0x7f13740711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 >> v1:172.21.15.77:6789/0 conn(0x7f1374154ac0 legacy=0x7f13740cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f137415b5f0 0x7f13600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1360003ef0 0x7f1360004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1360004d40 0x7f13600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f137406d790 con 0x7f1360004d40 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f137406da70 con 0x7f1360003ef0 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.777+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f136000b9f0 con 0x7f137415b5f0 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.778+0000 7f137ad1a640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1360004d40 0x7f13600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.778+0000 7f1379d18640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1360003ef0 0x7f1360004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.778+0000 7f137a519640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f137415b5f0 0x7f13600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.778+0000 7f137ad1a640 1 -- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f137415b5f0 msgr2=0x7f13600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.778+0000 7f137ad1a640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f137415b5f0 0x7f13600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.778+0000 7f137ad1a640 1 -- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1360003ef0 msgr2=0x7f1360004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.778+0000 7f137ad1a640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1360003ef0 0x7f1360004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.778+0000 7f137ad1a640 1 -- 172.21.15.77:0/4235748801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1360003520 con 0x7f1360004d40 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.778+0000 7f137ad1a640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1360004d40 0x7f13600051c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f136c009fc0 tx=0x7f1360005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb6c681b26c56f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.779+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c009c10 con 0x7f1360004d40 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.779+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f136c036070 con 0x7f1360004d40 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.779+0000 7f1379517640 1 -- 172.21.15.77:0/4235748801 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c02d070 con 0x7f1360004d40 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.779+0000 7f137c77d640 1 -- 172.21.15.77:0/4235748801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1360004d40 msgr2=0x7f13600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.779+0000 7f137c77d640 1 --2- 172.21.15.77:0/4235748801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1360004d40 0x7f13600051c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f136c009fc0 tx=0x7f1360005830 comp rx=0 tx=0).stop 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.779+0000 7f137ad1a640 1 -- 172.21.15.77:0/4235748801 reap_dead start 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.779+0000 7f137c77d640 1 -- 172.21.15.77:0/4235748801 shutdown_connections 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.779+0000 7f137c77d640 1 -- 172.21.15.77:0/4235748801 >> 172.21.15.77:0/4235748801 conn(0x7f13740b1820 msgr2=0x7f13740b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.779+0000 7f137c77d640 1 -- 172.21.15.77:0/4235748801 shutdown_connections 2024-04-25T01:24:11.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.779+0000 7f137c77d640 1 -- 172.21.15.77:0/4235748801 wait complete. 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.780+0000 7f137c77d640 1 Processor -- start 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.780+0000 7f137c77d640 1 -- start start 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137c77d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1360004d40 0x7f13741391e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137c77d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13741655a0 0x7f137414b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137c77d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1374165a20 0x7f137414b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137c77d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1374071370 con 0x7f1374165a20 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137c77d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f13740711f0 con 0x7f1360004d40 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137c77d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f13740714f0 con 0x7f13741655a0 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137ad1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1374165a20 0x7f137414b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137ad1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1374165a20 0x7f137414b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48632/0 (socket says 172.21.15.77:48632) 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137ad1a640 1 -- 172.21.15.77:0/2588087461 learned_addr learned my addr 172.21.15.77:0/2588087461 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137a519640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1360004d40 0x7f13741391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f1379d18640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13741655a0 0x7f137414b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137ad1a640 1 -- 172.21.15.77:0/2588087461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13741655a0 msgr2=0x7f137414b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137ad1a640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13741655a0 0x7f137414b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137ad1a640 1 -- 172.21.15.77:0/2588087461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1360004d40 msgr2=0x7f13741391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137ad1a640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1360004d40 0x7f13741391e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f137ad1a640 1 -- 172.21.15.77:0/2588087461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1374139720 con 0x7f1374165a20 2024-04-25T01:24:11.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.781+0000 7f1379d18640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13741655a0 0x7f137414b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:11.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.782+0000 7f137ad1a640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1374165a20 0x7f137414b6b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f136c033040 tx=0x7f136c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:11.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.782+0000 7f136affd640 1 -- 172.21.15.77:0/2588087461 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c009c10 con 0x7f1374165a20 2024-04-25T01:24:11.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.782+0000 7f136affd640 1 -- 172.21.15.77:0/2588087461 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f136c003b90 con 0x7f1374165a20 2024-04-25T01:24:11.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.782+0000 7f137c77d640 1 -- 172.21.15.77:0/2588087461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f137414cff0 con 0x7f1374165a20 2024-04-25T01:24:11.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.782+0000 7f136affd640 1 -- 172.21.15.77:0/2588087461 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c004020 con 0x7f1374165a20 2024-04-25T01:24:11.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.782+0000 7f137c77d640 1 -- 172.21.15.77:0/2588087461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f137414d290 con 0x7f1374165a20 2024-04-25T01:24:11.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.783+0000 7f137c77d640 1 -- 172.21.15.77:0/2588087461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1374068fb0 con 0x7f1374165a20 2024-04-25T01:24:11.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.783+0000 7f136affd640 1 -- 172.21.15.77:0/2588087461 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f136c005d10 con 0x7f1374165a20 2024-04-25T01:24:11.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.783+0000 7f136affd640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f135806be30 0x7f135806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:11.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.784+0000 7f136affd640 1 -- 172.21.15.77:0/2588087461 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6273+0+0 (secure 0 0 0) 0x7f136c0ac9f0 con 0x7f1374165a20 2024-04-25T01:24:11.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.784+0000 7f137a519640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f135806be30 0x7f135806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:11.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.784+0000 7f137a519640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f135806be30 0x7f135806e2f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1370001050 tx=0x7f1370009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:11.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.788+0000 7f136affd640 1 -- 172.21.15.77:0/2588087461 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f136c04a060 con 0x7f1374165a20 2024-04-25T01:24:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.997+0000 7f137c77d640 1 -- 172.21.15.77:0/2588087461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1374158390 con 0x7f1374165a20 2024-04-25T01:24:11.978 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-04-25T01:24:11.998+0000 7f46b1a53640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:24:11.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:11.999+0000 7f136affd640 1 -- 172.21.15.77:0/2588087461 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) ==== 161+0+0 (secure 0 0 0) 0x7f1374139720 con 0x7f1374165a20 2024-04-25T01:24:11.980 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T01:24:11.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 -- 172.21.15.77:0/2588087461 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f135806be30 msgr2=0x7f135806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f135806be30 0x7f135806e2f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1370001050 tx=0x7f1370009040 comp rx=0 tx=0).stop 2024-04-25T01:24:11.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 -- 172.21.15.77:0/2588087461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1374165a20 msgr2=0x7f137414b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:11.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1374165a20 0x7f137414b6b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f136c033040 tx=0x7f136c009750 comp rx=0 tx=0).stop 2024-04-25T01:24:11.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 -- 172.21.15.77:0/2588087461 shutdown_connections 2024-04-25T01:24:11.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f135806be30 0x7f135806e2f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1374165a20 0x7f137414b6b0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13741655a0 0x7f137414b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 --2- 172.21.15.77:0/2588087461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1360004d40 0x7f13741391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:11.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 -- 172.21.15.77:0/2588087461 >> 172.21.15.77:0/2588087461 conn(0x7f13740b1820 msgr2=0x7f13741516a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:11.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 -- 172.21.15.77:0/2588087461 shutdown_connections 2024-04-25T01:24:11.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.001+0000 7f137c77d640 1 -- 172.21.15.77:0/2588087461 wait complete. 2024-04-25T01:24:11.996 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T01:24:12.009 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-04-25T01:24:12.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.115+0000 7fcfc3c5d640 1 Processor -- start 2024-04-25T01:24:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.115+0000 7fcfc3c5d640 1 -- start start 2024-04-25T01:24:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc3c5d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcfbc158a70 0x7fcfbc0b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc3c5d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfbc0b37c0 0x7fcfbc0b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc3c5d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfbc15c210 con 0x7fcfbc15b520 2024-04-25T01:24:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc3c5d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fcfbc0b2ce0 con 0x7fcfbc158a70 2024-04-25T01:24:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc3c5d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcfbc0b2fc0 con 0x7fcfbc0b37c0 2024-04-25T01:24:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc19f9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcfbc158a70 0x7fcfbc0b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc21fa640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcfbc15b520 0x7fcfbc15b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49202/0 (socket says 172.21.15.77:49202) 2024-04-25T01:24:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc21fa640 1 -- 172.21.15.77:0/3249208226 learned_addr learned my addr 172.21.15.77:0/3249208226 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc11f8640 1 --2- 172.21.15.77:0/3249208226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfbc0b37c0 0x7fcfbc0b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc19f9640 1 -- 172.21.15.77:0/3249208226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfbc0b37c0 msgr2=0x7fcfbc0b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:12.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc19f9640 1 --2- 172.21.15.77:0/3249208226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfbc0b37c0 0x7fcfbc0b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc19f9640 1 -- 172.21.15.77:0/3249208226 >> v1:172.21.15.77:6789/0 conn(0x7fcfbc15b520 legacy=0x7fcfbc15b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:12.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.116+0000 7fcfc19f9640 1 -- 172.21.15.77:0/3249208226 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfbc0b32a0 con 0x7fcfbc158a70 2024-04-25T01:24:12.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.117+0000 7fcfc19f9640 1 --2- 172.21.15.77:0/3249208226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfbc158a70 0x7fcfbc0b40c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80097a0 tx=0x7fcfb802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61c97a0a5df791 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:12.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.117+0000 7fcfc09f7640 1 -- 172.21.15.77:0/3249208226 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8037070 con 0x7fcfbc158a70 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.117+0000 7fcfc09f7640 1 -- 172.21.15.77:0/3249208226 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfb8032440 con 0x7fcfbc158a70 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.117+0000 7fcfc09f7640 1 -- 172.21.15.77:0/3249208226 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8032810 con 0x7fcfbc158a70 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.117+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/3249208226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfbc158a70 msgr2=0x7fcfbc0b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.117+0000 7fcfc3c5d640 1 --2- 172.21.15.77:0/3249208226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfbc158a70 0x7fcfbc0b40c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80097a0 tx=0x7fcfb802a180 comp rx=0 tx=0).stop 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.117+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/3249208226 shutdown_connections 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.117+0000 7fcfc3c5d640 1 --2- 172.21.15.77:0/3249208226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfbc0b37c0 0x7fcfbc0b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.117+0000 7fcfc3c5d640 1 --2- 172.21.15.77:0/3249208226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfbc158a70 0x7fcfbc0b40c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.118+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/3249208226 >> 172.21.15.77:0/3249208226 conn(0x7fcfbc06b720 msgr2=0x7fcfbc0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.118+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/3249208226 shutdown_connections 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.118+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/3249208226 wait complete. 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.119+0000 7fcfc3c5d640 1 Processor -- start 2024-04-25T01:24:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.119+0000 7fcfc3c5d640 1 -- start start 2024-04-25T01:24:12.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.119+0000 7fcfc3c5d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfbc0b37c0 0x7fcfbc13dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.119+0000 7fcfc3c5d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfbc158a70 0x7fcfbc13e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.119+0000 7fcfc3c5d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfbc15b520 0x7fcfbc13b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.119+0000 7fcfc3c5d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc06cb70 con 0x7fcfbc15b520 2024-04-25T01:24:12.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.119+0000 7fcfc3c5d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc06c9f0 con 0x7fcfbc158a70 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.119+0000 7fcfc3c5d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcfbc06ccf0 con 0x7fcfbc0b37c0 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc21fa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfbc15b520 0x7fcfbc13b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc21fa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfbc15b520 0x7fcfbc13b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48648/0 (socket says 172.21.15.77:48648) 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc21fa640 1 -- 172.21.15.77:0/2520767413 learned_addr learned my addr 172.21.15.77:0/2520767413 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc11f8640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfbc158a70 0x7fcfbc13e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc19f9640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfbc0b37c0 0x7fcfbc13dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc21fa640 1 -- 172.21.15.77:0/2520767413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfbc0b37c0 msgr2=0x7fcfbc13dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc21fa640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfbc0b37c0 0x7fcfbc13dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc21fa640 1 -- 172.21.15.77:0/2520767413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfbc158a70 msgr2=0x7fcfbc13e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc21fa640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfbc158a70 0x7fcfbc13e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc21fa640 1 -- 172.21.15.77:0/2520767413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfbc13ed40 con 0x7fcfbc15b520 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc11f8640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfbc158a70 0x7fcfbc13e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc21fa640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfbc15b520 0x7fcfbc13b490 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400cab0 tx=0x7fcfb400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:12.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfb27fc640 1 -- 172.21.15.77:0/2520767413 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4017070 con 0x7fcfbc15b520 2024-04-25T01:24:12.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfb27fc640 1 -- 172.21.15.77:0/2520767413 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfb400ac30 con 0x7fcfbc15b520 2024-04-25T01:24:12.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfb27fc640 1 -- 172.21.15.77:0/2520767413 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4012400 con 0x7fcfbc15b520 2024-04-25T01:24:12.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/2520767413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfbc167dc0 con 0x7fcfbc15b520 2024-04-25T01:24:12.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.120+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/2520767413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfbc13d0c0 con 0x7fcfbc15b520 2024-04-25T01:24:12.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.121+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/2520767413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfbc068fb0 con 0x7fcfbc15b520 2024-04-25T01:24:12.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.122+0000 7fcfb27fc640 1 -- 172.21.15.77:0/2520767413 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcfb401a070 con 0x7fcfbc15b520 2024-04-25T01:24:12.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.122+0000 7fcfb27fc640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcfa006be30 0x7fcfa006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.122+0000 7fcfb27fc640 1 -- 172.21.15.77:0/2520767413 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6359+0+0 (secure 0 0 0) 0x7fcfb408ace0 con 0x7fcfbc15b520 2024-04-25T01:24:12.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.122+0000 7fcfc19f9640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcfa006be30 0x7fcfa006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.123+0000 7fcfc19f9640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcfa006be30 0x7fcfa006e2f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8034000 tx=0x7fcfb802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:12.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.127+0000 7fcfb27fc640 1 -- 172.21.15.77:0/2520767413 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcfbc13ed40 con 0x7fcfbc15b520 2024-04-25T01:24:12.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:12.336+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/2520767413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fcfbc165c80 con 0x7fcfbc15b520 2024-04-25T01:24:12.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.018+0000 7fcfb27fc640 1 -- 172.21.15.77:0/2520767413 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) ==== 104+0+0 (secure 0 0 0) 0x7fcfb4058db0 con 0x7fcfbc15b520 2024-04-25T01:24:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/2520767413 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcfa006be30 msgr2=0x7fcfa006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcfa006be30 0x7fcfa006e2f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8034000 tx=0x7fcfb802e000 comp rx=0 tx=0).stop 2024-04-25T01:24:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/2520767413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfbc15b520 msgr2=0x7fcfbc13b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfbc15b520 0x7fcfbc13b490 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400cab0 tx=0x7fcfb400cf70 comp rx=0 tx=0).stop 2024-04-25T01:24:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/2520767413 shutdown_connections 2024-04-25T01:24:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fcfa006be30 0x7fcfa006e2f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfbc15b520 0x7fcfbc13b490 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfbc158a70 0x7fcfbc13e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 --2- 172.21.15.77:0/2520767413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfbc0b37c0 0x7fcfbc13dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/2520767413 >> 172.21.15.77:0/2520767413 conn(0x7fcfbc06b720 msgr2=0x7fcfbc13f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/2520767413 shutdown_connections 2024-04-25T01:24:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.020+0000 7fcfc3c5d640 1 -- 172.21.15.77:0/2520767413 wait complete. 2024-04-25T01:24:13.013 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T01:24:13.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.177+0000 7fed0beba640 1 Processor -- start 2024-04-25T01:24:13.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.177+0000 7fed0beba640 1 -- start start 2024-04-25T01:24:13.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.178+0000 7fed0beba640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040d4a10 0x7fed040cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.178+0000 7fed0beba640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed040d1e90 con 0x7fed0415e4c0 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.178+0000 7fed0beba640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed040b54a0 con 0x7fed040b3ee0 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.178+0000 7fed0beba640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed040d2130 con 0x7fed040d4a10 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.178+0000 7fed09c56640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fed0415e4c0 0x7fed040d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49212/0 (socket says 172.21.15.77:49212) 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.178+0000 7fed09c56640 1 -- 172.21.15.77:0/1868430517 learned_addr learned my addr 172.21.15.77:0/1868430517 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.178+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3936909201 0 0) 0x7fed040d1e90 con 0x7fed0415e4c0 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.178+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecf0003520 con 0x7fed0415e4c0 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.178+0000 7fed09455640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040d4a10 0x7fed040cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2069848460 0 0) 0x7fecf0003520 con 0x7fed0415e4c0 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040d4a10 msgr2=0x7fed040cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040d4a10 0x7fed040cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 >> v1:172.21.15.121:6789/0 conn(0x7fed040b3ee0 legacy=0x7fed040d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed040b5780 con 0x7fed0415e4c0 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fed00001670 con 0x7fed0415e4c0 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 >> v1:172.21.15.77:6789/0 conn(0x7fed0415e4c0 legacy=0x7fed040d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf00039e0 0x7fecf0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf0004800 0x7fecf0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040d4a10 0x7fecf0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fed040d2130 con 0x7fecf00039e0 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fecf000baf0 con 0x7fecf0004800 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fecf000bcf0 con 0x7fed040d4a10 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed09c56640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf00039e0 0x7fecf0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed09455640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf0004800 0x7fecf0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed0a457640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040d4a10 0x7fecf0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed09c56640 1 -- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040d4a10 msgr2=0x7fecf0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed09c56640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040d4a10 0x7fecf0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed09c56640 1 -- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf0004800 msgr2=0x7fecf0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed09c56640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf0004800 0x7fecf0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed09c56640 1 -- 172.21.15.77:0/1868430517 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf0003520 con 0x7fecf00039e0 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.179+0000 7fed09455640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf0004800 0x7fecf0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.180+0000 7fed09c56640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf00039e0 0x7fecf0003e00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fed0000c210 tx=0x7fecf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22a4c3085398b0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.180+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00007d30 con 0x7fecf00039e0 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.180+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed00005ad0 con 0x7fecf00039e0 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.180+0000 7fed08c54640 1 -- 172.21.15.77:0/1868430517 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00007070 con 0x7fecf00039e0 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.180+0000 7fed0beba640 1 -- 172.21.15.77:0/1868430517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf00039e0 msgr2=0x7fecf0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.180+0000 7fed0beba640 1 --2- 172.21.15.77:0/1868430517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf00039e0 0x7fecf0003e00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fed0000c210 tx=0x7fecf0005830 comp rx=0 tx=0).stop 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.180+0000 7fed0a457640 1 -- 172.21.15.77:0/1868430517 reap_dead start 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.180+0000 7fed0beba640 1 -- 172.21.15.77:0/1868430517 shutdown_connections 2024-04-25T01:24:13.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.180+0000 7fed0beba640 1 -- 172.21.15.77:0/1868430517 >> 172.21.15.77:0/1868430517 conn(0x7fed040d3ef0 msgr2=0x7fed040d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:13.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.180+0000 7fed0beba640 1 -- 172.21.15.77:0/1868430517 shutdown_connections 2024-04-25T01:24:13.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.180+0000 7fed0beba640 1 -- 172.21.15.77:0/1868430517 wait complete. 2024-04-25T01:24:13.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed0beba640 1 Processor -- start 2024-04-25T01:24:13.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed0beba640 1 -- start start 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed0beba640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf00039e0 0x7fed04156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed0beba640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040c9120 0x7fed040c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed0beba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed040b6c60 0x7fed040b70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed0beba640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fed040b30a0 con 0x7fed040b6c60 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed0beba640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fed040b2f20 con 0x7fecf00039e0 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed0beba640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed0415e8a0 con 0x7fed040c9120 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed0a457640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed040b6c60 0x7fed040b70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed0a457640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed040b6c60 0x7fed040b70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48662/0 (socket says 172.21.15.77:48662) 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed0a457640 1 -- 172.21.15.77:0/2616371677 learned_addr learned my addr 172.21.15.77:0/2616371677 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed09c56640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf00039e0 0x7fed04156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.182+0000 7fed09455640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040c9120 0x7fed040c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.183+0000 7fed0a457640 1 -- 172.21.15.77:0/2616371677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040c9120 msgr2=0x7fed040c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.183+0000 7fed0a457640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040c9120 0x7fed040c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.183+0000 7fed0a457640 1 -- 172.21.15.77:0/2616371677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf00039e0 msgr2=0x7fed04156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.183+0000 7fed0a457640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf00039e0 0x7fed04156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.183+0000 7fed0a457640 1 -- 172.21.15.77:0/2616371677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed040b8550 con 0x7fed040b6c60 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.183+0000 7fed0a457640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed040b6c60 0x7fed040b70e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fed0406b940 tx=0x7fecfc00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.183+0000 7fecfa7fc640 1 -- 172.21.15.77:0/2616371677 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecfc003970 con 0x7fed040b6c60 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.183+0000 7fecfa7fc640 1 -- 172.21.15.77:0/2616371677 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecfc018070 con 0x7fed040b6c60 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.183+0000 7fed0beba640 1 -- 172.21.15.77:0/2616371677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed040b79d0 con 0x7fed040b6c60 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.183+0000 7fecfa7fc640 1 -- 172.21.15.77:0/2616371677 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecfc003d50 con 0x7fed040b6c60 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.183+0000 7fed0beba640 1 -- 172.21.15.77:0/2616371677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed040b7f90 con 0x7fed040b6c60 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.184+0000 7fed0beba640 1 -- 172.21.15.77:0/2616371677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed04068fb0 con 0x7fed040b6c60 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.184+0000 7fecfa7fc640 1 -- 172.21.15.77:0/2616371677 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fecfc0050d0 con 0x7fed040b6c60 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.185+0000 7fecfa7fc640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fece806be60 0x7fece806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.185+0000 7fecfa7fc640 1 -- 172.21.15.77:0/2616371677 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6359+0+0 (secure 0 0 0) 0x7fecfc08ab80 con 0x7fed040b6c60 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.185+0000 7fed09c56640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fece806be60 0x7fece806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.185+0000 7fed09c56640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fece806be60 0x7fece806e320 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fed00033000 tx=0x7fed0002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:13.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.189+0000 7fecfa7fc640 1 -- 172.21.15.77:0/2616371677 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fecfc058c50 con 0x7fed040b6c60 2024-04-25T01:24:13.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:13.361+0000 7fed0beba640 1 -- 172.21.15.77:0/2616371677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fed040ce220 con 0x7fed040b6c60 2024-04-25T01:24:14.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.027+0000 7fecfa7fc640 1 -- 172.21.15.77:0/2616371677 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v57) ==== 181+0+0 (secure 0 0 0) 0x7fecfc05cf50 con 0x7fed040b6c60 2024-04-25T01:24:14.008 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.030+0000 7fed0beba640 1 -- 172.21.15.77:0/2616371677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fece806be60 msgr2=0x7fece806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.030+0000 7fed0beba640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fece806be60 0x7fece806e320 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fed00033000 tx=0x7fed0002e000 comp rx=0 tx=0).stop 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.030+0000 7fed0beba640 1 -- 172.21.15.77:0/2616371677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed040b6c60 msgr2=0x7fed040b70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.030+0000 7fed0beba640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed040b6c60 0x7fed040b70e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fed0406b940 tx=0x7fecfc00eb90 comp rx=0 tx=0).stop 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.031+0000 7fed0beba640 1 -- 172.21.15.77:0/2616371677 shutdown_connections 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.031+0000 7fed0beba640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fece806be60 0x7fece806e320 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.031+0000 7fed0beba640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed040b6c60 0x7fed040b70e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.031+0000 7fed0beba640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed040c9120 0x7fed040c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.031+0000 7fed0beba640 1 --2- 172.21.15.77:0/2616371677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf00039e0 0x7fed04156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.031+0000 7fed0beba640 1 -- 172.21.15.77:0/2616371677 >> 172.21.15.77:0/2616371677 conn(0x7fed040d3ef0 msgr2=0x7fed040d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.031+0000 7fed0beba640 1 -- 172.21.15.77:0/2616371677 shutdown_connections 2024-04-25T01:24:14.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.031+0000 7fed0beba640 1 -- 172.21.15.77:0/2616371677 wait complete. 2024-04-25T01:24:14.027 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T01:24:14.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.151+0000 7f9a7fcf5640 1 Processor -- start 2024-04-25T01:24:14.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.151+0000 7f9a7fcf5640 1 -- start start 2024-04-25T01:24:14.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.152+0000 7f9a7fcf5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a7815ae90 0x7f9a780ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:14.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.152+0000 7f9a7fcf5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a780cf700 con 0x7f9a7806ea00 2024-04-25T01:24:14.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.152+0000 7f9a7fcf5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a780cd6b0 con 0x7f9a780ce470 2024-04-25T01:24:14.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.152+0000 7f9a7fcf5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f9a780cd990 con 0x7f9a7815ae90 2024-04-25T01:24:14.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.152+0000 7f9a7da91640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a7815ae90 0x7f9a780ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:14.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.152+0000 7f9a7e292640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9a7806ea00 0x7f9a78071470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49222/0 (socket says 172.21.15.77:49222) 2024-04-25T01:24:14.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.152+0000 7f9a7e292640 1 -- 172.21.15.77:0/4118161689 learned_addr learned my addr 172.21.15.77:0/4118161689 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:14.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.152+0000 7f9a7da91640 1 -- 172.21.15.77:0/4118161689 >> v1:172.21.15.121:6789/0 conn(0x7f9a780ce470 legacy=0x7f9a7806e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:14.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.152+0000 7f9a7da91640 1 -- 172.21.15.77:0/4118161689 >> v1:172.21.15.77:6789/0 conn(0x7f9a7806ea00 legacy=0x7f9a78071470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:14.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.152+0000 7f9a7da91640 1 -- 172.21.15.77:0/4118161689 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a780cdc70 con 0x7f9a7815ae90 2024-04-25T01:24:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.153+0000 7f9a7da91640 1 --2- 172.21.15.77:0/4118161689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a7815ae90 0x7f9a780ceb90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9a740097a0 tx=0x7f9a7402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7d6ec435f2ff64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.153+0000 7f9a7ca8f640 1 -- 172.21.15.77:0/4118161689 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74037070 con 0x7f9a7815ae90 2024-04-25T01:24:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.153+0000 7f9a7ca8f640 1 -- 172.21.15.77:0/4118161689 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a74032440 con 0x7f9a7815ae90 2024-04-25T01:24:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.153+0000 7f9a7ca8f640 1 -- 172.21.15.77:0/4118161689 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74032810 con 0x7f9a7815ae90 2024-04-25T01:24:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.154+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/4118161689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a7815ae90 msgr2=0x7f9a780ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.154+0000 7f9a7fcf5640 1 --2- 172.21.15.77:0/4118161689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a7815ae90 0x7f9a780ceb90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9a740097a0 tx=0x7f9a7402a230 comp rx=0 tx=0).stop 2024-04-25T01:24:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.154+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/4118161689 shutdown_connections 2024-04-25T01:24:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.154+0000 7f9a7fcf5640 1 --2- 172.21.15.77:0/4118161689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a7815ae90 0x7f9a780ceb90 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.154+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/4118161689 >> 172.21.15.77:0/4118161689 conn(0x7f9a780cfae0 msgr2=0x7f9a780cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.154+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/4118161689 shutdown_connections 2024-04-25T01:24:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.154+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/4118161689 wait complete. 2024-04-25T01:24:14.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.155+0000 7f9a7fcf5640 1 Processor -- start 2024-04-25T01:24:14.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.155+0000 7f9a7fcf5640 1 -- start start 2024-04-25T01:24:14.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.155+0000 7f9a7fcf5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a7806ea00 0x7f9a780b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:14.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.155+0000 7f9a7fcf5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a780ce470 0x7f9a780b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:14.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.155+0000 7f9a7fcf5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a7815ae90 0x7f9a78147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:14.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.155+0000 7f9a7fcf5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9a780cf0d0 con 0x7f9a780ce470 2024-04-25T01:24:14.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.155+0000 7f9a7fcf5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7806cf40 con 0x7f9a7806ea00 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.155+0000 7f9a7fcf5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a780cf250 con 0x7f9a7815ae90 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7d290640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a780ce470 0x7f9a780b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7d290640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a780ce470 0x7f9a780b5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48688/0 (socket says 172.21.15.77:48688) 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7d290640 1 -- 172.21.15.77:0/3963537418 learned_addr learned my addr 172.21.15.77:0/3963537418 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7da91640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a7806ea00 0x7f9a780b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7d290640 1 -- 172.21.15.77:0/3963537418 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a7815ae90 msgr2=0x7f9a78147450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7d290640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a7815ae90 0x7f9a78147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7d290640 1 -- 172.21.15.77:0/3963537418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a7806ea00 msgr2=0x7f9a780b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7d290640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a7806ea00 0x7f9a780b3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7d290640 1 -- 172.21.15.77:0/3963537418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a780b2c60 con 0x7f9a780ce470 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7d290640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a780ce470 0x7f9a780b5370 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800c430 tx=0x7f9a6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a6e7fc640 1 -- 172.21.15.77:0/3963537418 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a680052d0 con 0x7f9a780ce470 2024-04-25T01:24:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a6e7fc640 1 -- 172.21.15.77:0/3963537418 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a68005490 con 0x7f9a780ce470 2024-04-25T01:24:14.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/3963537418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a780b2f40 con 0x7f9a780ce470 2024-04-25T01:24:14.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a6e7fc640 1 -- 172.21.15.77:0/3963537418 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68007260 con 0x7f9a780ce470 2024-04-25T01:24:14.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.156+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/3963537418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a780b3500 con 0x7f9a780ce470 2024-04-25T01:24:14.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.157+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/3963537418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a78068fb0 con 0x7f9a780ce470 2024-04-25T01:24:14.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.157+0000 7f9a6e7fc640 1 -- 172.21.15.77:0/3963537418 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9a68007400 con 0x7f9a780ce470 2024-04-25T01:24:14.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.158+0000 7f9a6e7fc640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9a5c06be30 0x7f9a5c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:14.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.158+0000 7f9a7da91640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9a5c06be30 0x7f9a5c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:14.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.158+0000 7f9a6e7fc640 1 -- 172.21.15.77:0/3963537418 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5921+0+0 (secure 0 0 0) 0x7f9a6808a3d0 con 0x7f9a780ce470 2024-04-25T01:24:14.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.158+0000 7f9a7da91640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9a5c06be30 0x7f9a5c06e2f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9a74034000 tx=0x7f9a7402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:14.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.162+0000 7f9a6e7fc640 1 -- 172.21.15.77:0/3963537418 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9a68058d30 con 0x7f9a780ce470 2024-04-25T01:24:14.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:14.338+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/3963537418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9a7806ca90 con 0x7f9a780ce470 2024-04-25T01:24:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.033+0000 7f9a6e7fc640 1 -- 172.21.15.77:0/3963537418 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v58) ==== 169+0+0 (secure 0 0 0) 0x7f9a68058630 con 0x7f9a780ce470 2024-04-25T01:24:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-04-25T01:24:15.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.036+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/3963537418 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9a5c06be30 msgr2=0x7f9a5c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.036+0000 7f9a7fcf5640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9a5c06be30 0x7f9a5c06e2f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9a74034000 tx=0x7f9a7402e000 comp rx=0 tx=0).stop 2024-04-25T01:24:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.036+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/3963537418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a780ce470 msgr2=0x7f9a780b5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.036+0000 7f9a7fcf5640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a780ce470 0x7f9a780b5370 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800c430 tx=0x7f9a6800c8f0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.036+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/3963537418 shutdown_connections 2024-04-25T01:24:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.036+0000 7f9a7fcf5640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9a5c06be30 0x7f9a5c06e2f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.036+0000 7f9a7fcf5640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a780ce470 0x7f9a780b5370 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.036+0000 7f9a7fcf5640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a7815ae90 0x7f9a78147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.036+0000 7f9a7fcf5640 1 --2- 172.21.15.77:0/3963537418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a7806ea00 0x7f9a780b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.036+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/3963537418 >> 172.21.15.77:0/3963537418 conn(0x7f9a780cfae0 msgr2=0x7f9a780719f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.037+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/3963537418 shutdown_connections 2024-04-25T01:24:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.037+0000 7f9a7fcf5640 1 -- 172.21.15.77:0/3963537418 wait complete. 2024-04-25T01:24:15.030 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T01:24:15.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.151+0000 7f491a6b3640 1 Processor -- start 2024-04-25T01:24:15.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.151+0000 7f491a6b3640 1 -- start start 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.151+0000 7f491a6b3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f49140d27b0 0x7f49140d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.151+0000 7f491a6b3640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49140cea40 con 0x7f49140d23d0 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.151+0000 7f491a6b3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49140cece0 con 0x7f49140b9a90 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.151+0000 7f491a6b3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f49140cefc0 con 0x7f49140d27b0 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4913fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f49140d23d0 0x7f49140b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49238/0 (socket says 172.21.15.77:49238) 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4913fff640 1 -- 172.21.15.77:0/4029434737 learned_addr learned my addr 172.21.15.77:0/4029434737 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2096877939 0 0) 0x7f49140cea40 con 0x7f49140d23d0 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48fc003520 con 0x7f49140d23d0 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4918c50640 1 --2- 172.21.15.77:0/4029434737 >> v2:172.21.15.121:3300/0 conn(0x7f49140d27b0 0x7f49140d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3624040064 0 0) 0x7f48fc003520 con 0x7f49140d23d0 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 >> v2:172.21.15.121:3300/0 conn(0x7f49140d27b0 msgr2=0x7f49140d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 --2- 172.21.15.77:0/4029434737 >> v2:172.21.15.121:3300/0 conn(0x7f49140d27b0 0x7f49140d4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 >> v1:172.21.15.121:6789/0 conn(0x7f49140b9a90 legacy=0x7f49140b6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49140cf2a0 con 0x7f49140d23d0 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f4904001690 con 0x7f49140d23d0 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 >> v1:172.21.15.77:6789/0 conn(0x7f49140d23d0 legacy=0x7f49140b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 0x7f48fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48fc004800 0x7f48fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48fc0051c0 0x7f48fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49140cefc0 con 0x7f48fc0039b0 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f48fc00c0f0 con 0x7f48fc004800 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f48fc00c2f0 con 0x7f48fc0051c0 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4913fff640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 0x7f48fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f49137fe640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48fc004800 0x7f48fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4918c50640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48fc0051c0 0x7f48fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.152+0000 7f4913fff640 1 -- 172.21.15.77:0/4029434737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48fc0051c0 msgr2=0x7f48fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.153+0000 7f4913fff640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48fc0051c0 0x7f48fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.153+0000 7f4913fff640 1 -- 172.21.15.77:0/4029434737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48fc004800 msgr2=0x7f48fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.153+0000 7f4913fff640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48fc004800 0x7f48fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.153+0000 7f4913fff640 1 -- 172.21.15.77:0/4029434737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48fc003520 con 0x7f48fc0039b0 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.153+0000 7f4918c50640 1 -- 172.21.15.77:0/4029434737 reap_dead start 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.153+0000 7f49137fe640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48fc004800 0x7f48fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.153+0000 7f4913fff640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 0x7f48fc003dd0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f490400c210 tx=0x7f48fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea136674170cbbc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.153+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904007d10 con 0x7f48fc0039b0 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.153+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4904005ad0 con 0x7f48fc0039b0 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.153+0000 7f4912ffd640 1 -- 172.21.15.77:0/4029434737 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904007070 con 0x7f48fc0039b0 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.154+0000 7f491a6b3640 1 -- 172.21.15.77:0/4029434737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 msgr2=0x7f48fc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.154+0000 7f491a6b3640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 0x7f48fc003dd0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f490400c210 tx=0x7f48fc005830 comp rx=0 tx=0).stop 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.154+0000 7f491a6b3640 1 -- 172.21.15.77:0/4029434737 shutdown_connections 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.154+0000 7f491a6b3640 1 --2- 172.21.15.77:0/4029434737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 0x7f48fc003dd0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.154+0000 7f491a6b3640 1 -- 172.21.15.77:0/4029434737 >> 172.21.15.77:0/4029434737 conn(0x7f49140d3e30 msgr2=0x7f49140d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.154+0000 7f491a6b3640 1 -- 172.21.15.77:0/4029434737 shutdown_connections 2024-04-25T01:24:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.154+0000 7f491a6b3640 1 -- 172.21.15.77:0/4029434737 wait complete. 2024-04-25T01:24:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.155+0000 7f491a6b3640 1 Processor -- start 2024-04-25T01:24:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.155+0000 7f491a6b3640 1 -- start start 2024-04-25T01:24:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f491a6b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 0x7f491415b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f491a6b3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49140ca630 0x7f49140b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f491a6b3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49140b1ca0 0x7f491415e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f491a6b3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49140cfd80 con 0x7f48fc0039b0 2024-04-25T01:24:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f491a6b3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f49140cf860 con 0x7f49140b1ca0 2024-04-25T01:24:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f491a6b3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f49140d4920 con 0x7f49140ca630 2024-04-25T01:24:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f4913fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 0x7f491415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f4913fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 0x7f491415b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48716/0 (socket says 172.21.15.77:48716) 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f4913fff640 1 -- 172.21.15.77:0/1379156541 learned_addr learned my addr 172.21.15.77:0/1379156541 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f4913fff640 1 -- 172.21.15.77:0/1379156541 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49140ca630 msgr2=0x7f49140b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f4913fff640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49140ca630 0x7f49140b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f4913fff640 1 -- 172.21.15.77:0/1379156541 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49140b1ca0 msgr2=0x7f491415e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f4913fff640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49140b1ca0 0x7f491415e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f4913fff640 1 -- 172.21.15.77:0/1379156541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f491415c590 con 0x7f48fc0039b0 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f4913fff640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 0x7f491415b340 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f490400c210 tx=0x7f49140ceeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.156+0000 7f4910ff9640 1 -- 172.21.15.77:0/1379156541 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490403f070 con 0x7f48fc0039b0 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.157+0000 7f4910ff9640 1 -- 172.21.15.77:0/1379156541 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4904005c80 con 0x7f48fc0039b0 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.157+0000 7f491a6b3640 1 -- 172.21.15.77:0/1379156541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f491415c870 con 0x7f48fc0039b0 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.157+0000 7f4910ff9640 1 -- 172.21.15.77:0/1379156541 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904001650 con 0x7f48fc0039b0 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.157+0000 7f491a6b3640 1 -- 172.21.15.77:0/1379156541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f491415ce30 con 0x7f48fc0039b0 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.158+0000 7f491a6b3640 1 -- 172.21.15.77:0/1379156541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4914068fb0 con 0x7f48fc0039b0 2024-04-25T01:24:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.158+0000 7f4910ff9640 1 -- 172.21.15.77:0/1379156541 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4904042070 con 0x7f48fc0039b0 2024-04-25T01:24:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.158+0000 7f4910ff9640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f490006be30 0x7f490006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.159+0000 7f49137fe640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f490006be30 0x7f490006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.159+0000 7f4910ff9640 1 -- 172.21.15.77:0/1379156541 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f49040aaa90 con 0x7f48fc0039b0 2024-04-25T01:24:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.159+0000 7f49137fe640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f490006be30 0x7f490006e2f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f49141617b0 tx=0x7f4908009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.163+0000 7f4910ff9640 1 -- 172.21.15.77:0/1379156541 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4904049030 con 0x7f48fc0039b0 2024-04-25T01:24:15.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.337+0000 7f491a6b3640 1 -- 172.21.15.77:0/1379156541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f4914068fb0 con 0x7f48fc0039b0 2024-04-25T01:24:15.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.340+0000 7f4910ff9640 1 -- 172.21.15.77:0/1379156541 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7f4904042e40 con 0x7f48fc0039b0 2024-04-25T01:24:15.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 -- 172.21.15.77:0/1379156541 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f490006be30 msgr2=0x7f490006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f490006be30 0x7f490006e2f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f49141617b0 tx=0x7f4908009040 comp rx=0 tx=0).stop 2024-04-25T01:24:15.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 -- 172.21.15.77:0/1379156541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 msgr2=0x7f491415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 0x7f491415b340 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f490400c210 tx=0x7f49140ceeb0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 -- 172.21.15.77:0/1379156541 shutdown_connections 2024-04-25T01:24:15.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f490006be30 0x7f490006e2f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49140b1ca0 0x7f491415e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49140ca630 0x7f49140b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 --2- 172.21.15.77:0/1379156541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0039b0 0x7f491415b340 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 -- 172.21.15.77:0/1379156541 >> 172.21.15.77:0/1379156541 conn(0x7f49140d3e30 msgr2=0x7f49140cd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 -- 172.21.15.77:0/1379156541 shutdown_connections 2024-04-25T01:24:15.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.342+0000 7f491a6b3640 1 -- 172.21.15.77:0/1379156541 wait complete. 2024-04-25T01:24:15.334 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_mds_readonly' 2024-04-25T01:24:15.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.450+0000 7f31d9aa5640 1 Processor -- start 2024-04-25T01:24:15.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.450+0000 7f31d9aa5640 1 -- start start 2024-04-25T01:24:15.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.450+0000 7f31d9aa5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f31d40d4a10 0x7f31d40cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.450+0000 7f31d9aa5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31d40b3ee0 0x7f31d40b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.450+0000 7f31d9aa5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31d40d1e90 con 0x7f31d415e4c0 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.450+0000 7f31d9aa5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f31d40b54a0 con 0x7f31d40d4a10 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.450+0000 7f31d9aa5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f31d40d2130 con 0x7f31d40b3ee0 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.451+0000 7f31cbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f31d40d4a10 0x7f31d40cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.451+0000 7f31cbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f31d40d4a10 0x7f31d40cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48728/0 (socket says 172.21.15.77:48728) 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.451+0000 7f31cbfff640 1 -- 172.21.15.77:0/3287554929 learned_addr learned my addr 172.21.15.77:0/3287554929 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.451+0000 7f31d3fff640 1 --2- 172.21.15.77:0/3287554929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31d40b3ee0 0x7f31d40b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.451+0000 7f31cbfff640 1 -- 172.21.15.77:0/3287554929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31d40b3ee0 msgr2=0x7f31d40b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.451+0000 7f31cbfff640 1 --2- 172.21.15.77:0/3287554929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31d40b3ee0 0x7f31d40b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.451+0000 7f31cbfff640 1 -- 172.21.15.77:0/3287554929 >> v1:172.21.15.121:6789/0 conn(0x7f31d415e4c0 legacy=0x7f31d40d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.451+0000 7f31cbfff640 1 -- 172.21.15.77:0/3287554929 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d40b5780 con 0x7f31d40d4a10 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31cbfff640 1 --2- 172.21.15.77:0/3287554929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d40d4a10 0x7f31d40cfaf0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f31bc009870 tx=0x7f31bc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06ab91e7b968e68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31d2ffd640 1 -- 172.21.15.77:0/3287554929 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc037070 con 0x7f31d40d4a10 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31d2ffd640 1 -- 172.21.15.77:0/3287554929 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31bc032440 con 0x7f31d40d4a10 2024-04-25T01:24:15.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31d2ffd640 1 -- 172.21.15.77:0/3287554929 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc032810 con 0x7f31d40d4a10 2024-04-25T01:24:15.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31d9aa5640 1 -- 172.21.15.77:0/3287554929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d40d4a10 msgr2=0x7f31d40cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31d9aa5640 1 --2- 172.21.15.77:0/3287554929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d40d4a10 0x7f31d40cfaf0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f31bc009870 tx=0x7f31bc02a1c0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31d9aa5640 1 -- 172.21.15.77:0/3287554929 shutdown_connections 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31d9aa5640 1 --2- 172.21.15.77:0/3287554929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31d40b3ee0 0x7f31d40b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31d9aa5640 1 --2- 172.21.15.77:0/3287554929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d40d4a10 0x7f31d40cfaf0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31d9aa5640 1 -- 172.21.15.77:0/3287554929 >> 172.21.15.77:0/3287554929 conn(0x7f31d40d3ef0 msgr2=0x7f31d40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31d9aa5640 1 -- 172.21.15.77:0/3287554929 shutdown_connections 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.452+0000 7f31d9aa5640 1 -- 172.21.15.77:0/3287554929 wait complete. 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.453+0000 7f31d9aa5640 1 Processor -- start 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31d9aa5640 1 -- start start 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31d9aa5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31d40b3ee0 0x7f31d41561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31d9aa5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31d40d4a10 0x7f31d40b8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31d9aa5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d415e4c0 0x7f31d40b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31d9aa5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f31d40b30a0 con 0x7f31d415e4c0 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31d9aa5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f31d40b2f20 con 0x7f31d40b3ee0 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31d9aa5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f31d415e8a0 con 0x7f31d40d4a10 2024-04-25T01:24:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31d3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d415e4c0 0x7f31d40b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31d3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d415e4c0 0x7f31d40b6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48742/0 (socket says 172.21.15.77:48742) 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31d3fff640 1 -- 172.21.15.77:0/153925895 learned_addr learned my addr 172.21.15.77:0/153925895 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31d37fe640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31d40b3ee0 0x7f31d41561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.454+0000 7f31cbfff640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31d40d4a10 0x7f31d40b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.455+0000 7f31d3fff640 1 -- 172.21.15.77:0/153925895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31d40d4a10 msgr2=0x7f31d40b8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.455+0000 7f31d3fff640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31d40d4a10 0x7f31d40b8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.455+0000 7f31d3fff640 1 -- 172.21.15.77:0/153925895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31d40b3ee0 msgr2=0x7f31d41561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.455+0000 7f31d3fff640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31d40b3ee0 0x7f31d41561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.455+0000 7f31d3fff640 1 -- 172.21.15.77:0/153925895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d40b74f0 con 0x7f31d415e4c0 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.455+0000 7f31d3fff640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d415e4c0 0x7f31d40b6b10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f31cc007e40 tx=0x7f31cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.455+0000 7f31d0ff9640 1 -- 172.21.15.77:0/153925895 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc014620 con 0x7f31d415e4c0 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.455+0000 7f31d0ff9640 1 -- 172.21.15.77:0/153925895 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc019070 con 0x7f31d415e4c0 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.455+0000 7f31d9aa5640 1 -- 172.21.15.77:0/153925895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d40b77d0 con 0x7f31d415e4c0 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.455+0000 7f31d0ff9640 1 -- 172.21.15.77:0/153925895 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc014a20 con 0x7f31d415e4c0 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.455+0000 7f31d9aa5640 1 -- 172.21.15.77:0/153925895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d40b7d90 con 0x7f31d415e4c0 2024-04-25T01:24:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.456+0000 7f31d9aa5640 1 -- 172.21.15.77:0/153925895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d4068fb0 con 0x7f31d415e4c0 2024-04-25T01:24:15.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.456+0000 7f31d0ff9640 1 -- 172.21.15.77:0/153925895 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f31cc014bc0 con 0x7f31d415e4c0 2024-04-25T01:24:15.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.457+0000 7f31d0ff9640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f31b406be30 0x7f31b406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:15.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.457+0000 7f31d37fe640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f31b406be30 0x7f31b406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:15.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.457+0000 7f31d0ff9640 1 -- 172.21.15.77:0/153925895 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f31cc08b4d0 con 0x7f31d415e4c0 2024-04-25T01:24:15.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.457+0000 7f31d37fe640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f31b406be30 0x7f31b406e2f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f31c40016e0 tx=0x7f31c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:15.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.461+0000 7f31d0ff9640 1 -- 172.21.15.77:0/153925895 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f31cc059900 con 0x7f31d415e4c0 2024-04-25T01:24:15.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:15.702+0000 7f31d9aa5640 1 -- 172.21.15.77:0/153925895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_mds_readonly"]} v 0) -- 0x7f31d40b3920 con 0x7f31d415e4c0 2024-04-25T01:24:15.965 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:24:15.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:24:16.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.037+0000 7f31d0ff9640 1 -- 172.21.15.77:0/153925895 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_auto_repair.TestMDSAutoRepair.test_mds_readonly"]}]=0 v158) ==== 144+0+0 (secure 0 0 0) 0x7f31cc05dc00 con 0x7f31d415e4c0 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.039+0000 7f31d9aa5640 1 -- 172.21.15.77:0/153925895 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f31b406be30 msgr2=0x7f31b406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.039+0000 7f31d9aa5640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f31b406be30 0x7f31b406e2f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f31c40016e0 tx=0x7f31c4009040 comp rx=0 tx=0).stop 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.039+0000 7f31d9aa5640 1 -- 172.21.15.77:0/153925895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d415e4c0 msgr2=0x7f31d40b6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.039+0000 7f31d9aa5640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d415e4c0 0x7f31d40b6b10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f31cc007e40 tx=0x7f31cc00e4b0 comp rx=0 tx=0).stop 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.039+0000 7f31d9aa5640 1 -- 172.21.15.77:0/153925895 shutdown_connections 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.039+0000 7f31d9aa5640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f31b406be30 0x7f31b406e2f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.039+0000 7f31d9aa5640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d415e4c0 0x7f31d40b6b10 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.039+0000 7f31d9aa5640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31d40d4a10 0x7f31d40b8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.039+0000 7f31d9aa5640 1 --2- 172.21.15.77:0/153925895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31d40b3ee0 0x7f31d41561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.039+0000 7f31d9aa5640 1 -- 172.21.15.77:0/153925895 >> 172.21.15.77:0/153925895 conn(0x7f31d40d3ef0 msgr2=0x7f31d40b4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.039+0000 7f31d9aa5640 1 -- 172.21.15.77:0/153925895 shutdown_connections 2024-04-25T01:24:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:16.040+0000 7f31d9aa5640 1 -- 172.21.15.77:0/153925895 wait complete. 2024-04-25T01:24:16.031 INFO:tasks.cephfs_test_runner:test if MDS behave correct when it's readonly ... ok 2024-04-25T01:24:16.032 INFO:tasks.cephfs_test_runner: 2024-04-25T01:24:16.032 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-25T01:24:16.032 INFO:tasks.cephfs_test_runner:Ran 2 tests in 104.299s 2024-04-25T01:24:16.032 INFO:tasks.cephfs_test_runner: 2024-04-25T01:24:16.032 INFO:tasks.cephfs_test_runner:OK 2024-04-25T01:24:16.033 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-25T01:24:16.042 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T01:24:16.101 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T01:24:16.101 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:24:16.101 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:24:16.158 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T01:24:16.159 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-25T01:24:16.159 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:24:16.216 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T01:24:16.216 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:24:16.216 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:24:16.235 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=434 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:24:16.235 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-25T01:24:16.235 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:24:16.293 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=434 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:24:16.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.293 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-04-25T01:24:16.311 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T01:24:16.312 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:24:16.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.315 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T01:24:16.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.377 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:24:16.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.443 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-04-25T01:24:16.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.532 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-04-25T01:24:16.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.549 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-04-25T01:24:16.616 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2024-04-25T01:24:16.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:24:16.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.618 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-25T01:24:16.636 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T01:24:16.636 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T01:24:16.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T01:24:16.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T01:24:16.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2024-04-25T01:24:16.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.862 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-25T01:24:16.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:24:16.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2024-04-25T01:24:16.944 INFO:teuthology.orchestra.run.smithi121.stderr:Cannot find device "ceph-brx" 2024-04-25T01:24:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:24:16.946 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T01:24:16.964 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T01:24:16.964 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:24:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.082+0000 7f37714d4640 1 Processor -- start 2024-04-25T01:24:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.082+0000 7f37714d4640 1 -- start start 2024-04-25T01:24:17.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.082+0000 7f37714d4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f376c154ac0 0x7f376c15b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.082+0000 7f37714d4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f376c06e550 0x7f376c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.082+0000 7f37714d4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f376c154ea0 con 0x7f376c0cf2a0 2024-04-25T01:24:17.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.082+0000 7f37714d4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f376c06eee0 con 0x7f376c06e550 2024-04-25T01:24:17.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.082+0000 7f37714d4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f376c06d540 con 0x7f376c154ac0 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.083+0000 7f376b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f376c06e550 0x7f376c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.083+0000 7f376b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f376c06e550 0x7f376c06e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49604/0 (socket says 172.21.15.77:49604) 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.083+0000 7f376b7fe640 1 -- 172.21.15.77:0/3773667679 learned_addr learned my addr 172.21.15.77:0/3773667679 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.083+0000 7f376affd640 1 --2- 172.21.15.77:0/3773667679 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f376c154ac0 0x7f376c15b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.083+0000 7f376b7fe640 1 -- 172.21.15.77:0/3773667679 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f376c154ac0 msgr2=0x7f376c15b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.083+0000 7f376b7fe640 1 --2- 172.21.15.77:0/3773667679 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f376c154ac0 0x7f376c15b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.083+0000 7f376b7fe640 1 -- 172.21.15.77:0/3773667679 >> v1:172.21.15.121:6789/0 conn(0x7f376c0cf2a0 legacy=0x7f376c0711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.083+0000 7f376b7fe640 1 -- 172.21.15.77:0/3773667679 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c06d790 con 0x7f376c06e550 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.083+0000 7f376affd640 1 --2- 172.21.15.77:0/3773667679 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f376c154ac0 0x7f376c15b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f376b7fe640 1 --2- 172.21.15.77:0/3773667679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f376c06e550 0x7f376c06e9a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3764009e50 tx=0x7f376402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc53ff116be8574 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f3769ffb640 1 -- 172.21.15.77:0/3773667679 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764036070 con 0x7f376c06e550 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f3769ffb640 1 -- 172.21.15.77:0/3773667679 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3764031600 con 0x7f376c06e550 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f3769ffb640 1 -- 172.21.15.77:0/3773667679 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37640319d0 con 0x7f376c06e550 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f37714d4640 1 -- 172.21.15.77:0/3773667679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f376c06e550 msgr2=0x7f376c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f37714d4640 1 --2- 172.21.15.77:0/3773667679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f376c06e550 0x7f376c06e9a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3764009e50 tx=0x7f376402a710 comp rx=0 tx=0).stop 2024-04-25T01:24:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f37714d4640 1 -- 172.21.15.77:0/3773667679 shutdown_connections 2024-04-25T01:24:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f37714d4640 1 --2- 172.21.15.77:0/3773667679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f376c06e550 0x7f376c06e9a0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f37714d4640 1 --2- 172.21.15.77:0/3773667679 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f376c154ac0 0x7f376c15b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f37714d4640 1 -- 172.21.15.77:0/3773667679 >> 172.21.15.77:0/3773667679 conn(0x7f376c0b1820 msgr2=0x7f376c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f37714d4640 1 -- 172.21.15.77:0/3773667679 shutdown_connections 2024-04-25T01:24:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.084+0000 7f37714d4640 1 -- 172.21.15.77:0/3773667679 wait complete. 2024-04-25T01:24:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.085+0000 7f37714d4640 1 Processor -- start 2024-04-25T01:24:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f37714d4640 1 -- start start 2024-04-25T01:24:17.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f37714d4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f376c06e550 0x7f376c0b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f37714d4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f376c0cf2a0 0x7f376c1430f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f37714d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f376c154ac0 0x7f376c143630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f37714d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f376c06e030 con 0x7f376c154ac0 2024-04-25T01:24:17.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f37714d4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f376c06dd50 con 0x7f376c06e550 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f37714d4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f376c071900 con 0x7f376c0cf2a0 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f376c154ac0 0x7f376c143630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f376c154ac0 0x7f376c143630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49606/0 (socket says 172.21.15.77:49606) 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f376b7fe640 1 -- 172.21.15.77:0/2964632014 learned_addr learned my addr 172.21.15.77:0/2964632014 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f376affd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f376c06e550 0x7f376c0b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.086+0000 7f376a7fc640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f376c0cf2a0 0x7f376c1430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.087+0000 7f376b7fe640 1 -- 172.21.15.77:0/2964632014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f376c0cf2a0 msgr2=0x7f376c1430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.087+0000 7f376b7fe640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f376c0cf2a0 0x7f376c1430f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.087+0000 7f376b7fe640 1 -- 172.21.15.77:0/2964632014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f376c06e550 msgr2=0x7f376c0b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.087+0000 7f376b7fe640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f376c06e550 0x7f376c0b3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.087+0000 7f376b7fe640 1 -- 172.21.15.77:0/2964632014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c0b08a0 con 0x7f376c154ac0 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.087+0000 7f376b7fe640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f376c154ac0 0x7f376c143630 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3764034040 tx=0x7f3764004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.087+0000 7f37537fe640 1 -- 172.21.15.77:0/2964632014 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764036050 con 0x7f376c154ac0 2024-04-25T01:24:17.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.087+0000 7f37537fe640 1 -- 172.21.15.77:0/2964632014 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3764042070 con 0x7f376c154ac0 2024-04-25T01:24:17.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.087+0000 7f37714d4640 1 -- 172.21.15.77:0/2964632014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c0b0b60 con 0x7f376c154ac0 2024-04-25T01:24:17.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.087+0000 7f37537fe640 1 -- 172.21.15.77:0/2964632014 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764009510 con 0x7f376c154ac0 2024-04-25T01:24:17.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.087+0000 7f37714d4640 1 -- 172.21.15.77:0/2964632014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c0b1120 con 0x7f376c154ac0 2024-04-25T01:24:17.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.088+0000 7f37714d4640 1 -- 172.21.15.77:0/2964632014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c068fb0 con 0x7f376c154ac0 2024-04-25T01:24:17.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.089+0000 7f37537fe640 1 -- 172.21.15.77:0/2964632014 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3764007bd0 con 0x7f376c154ac0 2024-04-25T01:24:17.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.089+0000 7f37537fe640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f375806be30 0x7f375806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.089+0000 7f37537fe640 1 -- 172.21.15.77:0/2964632014 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f37640abb20 con 0x7f376c154ac0 2024-04-25T01:24:17.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.089+0000 7f376affd640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f375806be30 0x7f375806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.089+0000 7f376affd640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f375806be30 0x7f375806e2f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3760006fd0 tx=0x7f3760008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.093+0000 7f37537fe640 1 -- 172.21.15.77:0/2964632014 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3764079eb0 con 0x7f376c154ac0 2024-04-25T01:24:17.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.262+0000 7f37714d4640 1 -- 172.21.15.77:0/2964632014 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f376c15b3d0 con 0x7f375806be30 2024-04-25T01:24:17.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.263+0000 7f37537fe640 1 -- 172.21.15.77:0/2964632014 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13419 (secure 0 0 0) 0x7f376c15b3d0 con 0x7f375806be30 2024-04-25T01:24:17.244 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:17.244 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:24:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.264+0000 7f37714d4640 1 -- 172.21.15.77:0/2964632014 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f375806be30 msgr2=0x7f375806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.264+0000 7f37714d4640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f375806be30 0x7f375806e2f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3760006fd0 tx=0x7f3760008040 comp rx=0 tx=0).stop 2024-04-25T01:24:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.264+0000 7f37714d4640 1 -- 172.21.15.77:0/2964632014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f376c154ac0 msgr2=0x7f376c143630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.264+0000 7f37714d4640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f376c154ac0 0x7f376c143630 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3764034040 tx=0x7f3764004a40 comp rx=0 tx=0).stop 2024-04-25T01:24:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.265+0000 7f37714d4640 1 -- 172.21.15.77:0/2964632014 shutdown_connections 2024-04-25T01:24:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.265+0000 7f37714d4640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f375806be30 0x7f375806e2f0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.265+0000 7f37714d4640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f376c154ac0 0x7f376c143630 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.265+0000 7f37714d4640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f376c0cf2a0 0x7f376c1430f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.265+0000 7f37714d4640 1 --2- 172.21.15.77:0/2964632014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f376c06e550 0x7f376c0b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.265+0000 7f37714d4640 1 -- 172.21.15.77:0/2964632014 >> 172.21.15.77:0/2964632014 conn(0x7f376c0b1820 msgr2=0x7f376c072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.265+0000 7f37714d4640 1 -- 172.21.15.77:0/2964632014 shutdown_connections 2024-04-25T01:24:17.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.265+0000 7f37714d4640 1 -- 172.21.15.77:0/2964632014 wait complete. 2024-04-25T01:24:17.256 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-25T01:24:17.186850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222600,"kb_used_data":6432,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752120,"statfs":{"total":773094113280,"available":772866170880,"internally_reserved":0,"allocated":6586368,"data_stored":3616920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17778,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-80,"num_read_kb":-94,"num_write":-66,"num_write_kb":-40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000830"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":235,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T01:24:01.954837+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:24:01.954837+0000","last_peered":"2024-04-25T01:24:01.954837+0000","last_clean":"2024-04-25T01:24:01.954837+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:24:01.954837+0000","last_undegraded":"2024-04-25T01:24:01.954837+0000","last_fullsized":"2024-04-25T01:24:01.954837+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1257472,"data_stored":886636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":303441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":655360,"data_stored":296144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":659456,"data_stored":296726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":692224,"data_stored":315594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":322175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":302991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1269760,"data_stored":893213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:24:17.257 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:24:17.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.372+0000 7f973279e640 1 Processor -- start 2024-04-25T01:24:17.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.373+0000 7f973279e640 1 -- start start 2024-04-25T01:24:17.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.373+0000 7f973279e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f972c15df10 0x7f972c06f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.373+0000 7f973279e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f972c0b57c0 0x7f972c06b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.373+0000 7f973279e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f972c06b660 0x7f972c161b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.373+0000 7f973279e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f972c06bac0 con 0x7f972c06b660 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.373+0000 7f973279e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f972c0b5c20 con 0x7f972c0b57c0 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.373+0000 7f973279e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f972c071d40 con 0x7f972c15df10 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.373+0000 7f9731f9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f972c06b660 0x7f972c161b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.373+0000 7f9731f9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f972c06b660 0x7f972c161b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49612/0 (socket says 172.21.15.77:49612) 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.374+0000 7f9731f9d640 1 -- 172.21.15.77:0/1485009309 learned_addr learned my addr 172.21.15.77:0/1485009309 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.374+0000 7f9730f9b640 1 --2- 172.21.15.77:0/1485009309 >> v2:172.21.15.121:3300/0 conn(0x7f972c0b57c0 0x7f972c06b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.374+0000 7f973179c640 1 --2- 172.21.15.77:0/1485009309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f972c15df10 0x7f972c06f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.374+0000 7f9731f9d640 1 -- 172.21.15.77:0/1485009309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f972c15df10 msgr2=0x7f972c06f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.374+0000 7f9731f9d640 1 --2- 172.21.15.77:0/1485009309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f972c15df10 0x7f972c06f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.374+0000 7f9731f9d640 1 -- 172.21.15.77:0/1485009309 >> v2:172.21.15.121:3300/0 conn(0x7f972c0b57c0 msgr2=0x7f972c06b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.374+0000 7f9731f9d640 1 --2- 172.21.15.77:0/1485009309 >> v2:172.21.15.121:3300/0 conn(0x7f972c0b57c0 0x7f972c06b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.374+0000 7f9731f9d640 1 -- 172.21.15.77:0/1485009309 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f972c072020 con 0x7f972c06b660 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.374+0000 7f9731f9d640 1 --2- 172.21.15.77:0/1485009309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f972c06b660 0x7f972c161b30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f972c067c70 tx=0x7f972402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f85bf6aedf24a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f9723fff640 1 -- 172.21.15.77:0/1485009309 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9724037070 con 0x7f972c06b660 2024-04-25T01:24:17.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f9723fff640 1 -- 172.21.15.77:0/1485009309 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9724032440 con 0x7f972c06b660 2024-04-25T01:24:17.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f9723fff640 1 -- 172.21.15.77:0/1485009309 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9724032810 con 0x7f972c06b660 2024-04-25T01:24:17.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f973279e640 1 -- 172.21.15.77:0/1485009309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f972c06b660 msgr2=0x7f972c161b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f973279e640 1 --2- 172.21.15.77:0/1485009309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f972c06b660 0x7f972c161b30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f972c067c70 tx=0x7f972402a1e0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f973279e640 1 -- 172.21.15.77:0/1485009309 shutdown_connections 2024-04-25T01:24:17.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f973279e640 1 --2- 172.21.15.77:0/1485009309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f972c06b660 0x7f972c161b30 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f973279e640 1 --2- 172.21.15.77:0/1485009309 >> v2:172.21.15.121:3300/0 conn(0x7f972c0b57c0 0x7f972c06b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f973279e640 1 --2- 172.21.15.77:0/1485009309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f972c15df10 0x7f972c06f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f973279e640 1 -- 172.21.15.77:0/1485009309 >> 172.21.15.77:0/1485009309 conn(0x7f972c0d0a40 msgr2=0x7f972c0d0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f973279e640 1 -- 172.21.15.77:0/1485009309 shutdown_connections 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.375+0000 7f973279e640 1 -- 172.21.15.77:0/1485009309 wait complete. 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f973279e640 1 Processor -- start 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f973279e640 1 -- start start 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f973279e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f972c06b660 0x7f972c158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f973279e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f972c0b57c0 0x7f972c0c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f973279e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c15df10 0x7f972c0c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f973279e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f972c0afd20 con 0x7f972c06b660 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f973279e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f972c0725e0 con 0x7f972c15df10 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f973279e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f972c15e2f0 con 0x7f972c0b57c0 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f973179c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f972c06b660 0x7f972c158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f973179c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f972c06b660 0x7f972c158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49614/0 (socket says 172.21.15.77:49614) 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f973179c640 1 -- 172.21.15.77:0/1789960767 learned_addr learned my addr 172.21.15.77:0/1789960767 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f9730f9b640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f972c0b57c0 0x7f972c0c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.377+0000 7f9731f9d640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c15df10 0x7f972c0c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.378+0000 7f973179c640 1 -- 172.21.15.77:0/1789960767 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f972c0b57c0 msgr2=0x7f972c0c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.378+0000 7f973179c640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f972c0b57c0 0x7f972c0c6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.378+0000 7f973179c640 1 -- 172.21.15.77:0/1789960767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c15df10 msgr2=0x7f972c0c59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.378+0000 7f973179c640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c15df10 0x7f972c0c59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.378+0000 7f973179c640 1 -- 172.21.15.77:0/1789960767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f972c15a7b0 con 0x7f972c06b660 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.378+0000 7f973179c640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f972c06b660 0x7f972c158300 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f972800efc0 tx=0x7f972800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.378+0000 7f9721ffb640 1 -- 172.21.15.77:0/1789960767 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9728018020 con 0x7f972c06b660 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.378+0000 7f9721ffb640 1 -- 172.21.15.77:0/1789960767 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9728014480 con 0x7f972c06b660 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.378+0000 7f973279e640 1 -- 172.21.15.77:0/1789960767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f972c15aaf0 con 0x7f972c06b660 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.378+0000 7f9721ffb640 1 -- 172.21.15.77:0/1789960767 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9728014850 con 0x7f972c06b660 2024-04-25T01:24:17.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.378+0000 7f973279e640 1 -- 172.21.15.77:0/1789960767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f972c15b330 con 0x7f972c06b660 2024-04-25T01:24:17.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.379+0000 7f973279e640 1 -- 172.21.15.77:0/1789960767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f972c075350 con 0x7f972c06b660 2024-04-25T01:24:17.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.380+0000 7f9721ffb640 1 -- 172.21.15.77:0/1789960767 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9728005d10 con 0x7f972c06b660 2024-04-25T01:24:17.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.380+0000 7f9721ffb640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f971006be30 0x7f971006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.380+0000 7f9721ffb640 1 -- 172.21.15.77:0/1789960767 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f972808b700 con 0x7f972c06b660 2024-04-25T01:24:17.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.380+0000 7f9730f9b640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f971006be30 0x7f971006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.380+0000 7f9730f9b640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f971006be30 0x7f971006e2f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f972c06b400 tx=0x7f971c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.384+0000 7f9721ffb640 1 -- 172.21.15.77:0/1789960767 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9728022070 con 0x7f972c06b660 2024-04-25T01:24:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.553+0000 7f973279e640 1 -- 172.21.15.77:0/1789960767 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f972c0b0c70 con 0x7f971006be30 2024-04-25T01:24:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.554+0000 7f9721ffb640 1 -- 172.21.15.77:0/1789960767 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13419 (secure 0 0 0) 0x7f972c0b0c70 con 0x7f971006be30 2024-04-25T01:24:17.535 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:17.535 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:24:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 -- 172.21.15.77:0/1789960767 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f971006be30 msgr2=0x7f971006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f971006be30 0x7f971006e2f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f972c06b400 tx=0x7f971c009190 comp rx=0 tx=0).stop 2024-04-25T01:24:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 -- 172.21.15.77:0/1789960767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f972c06b660 msgr2=0x7f972c158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f972c06b660 0x7f972c158300 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f972800efc0 tx=0x7f972800c490 comp rx=0 tx=0).stop 2024-04-25T01:24:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 -- 172.21.15.77:0/1789960767 shutdown_connections 2024-04-25T01:24:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f971006be30 0x7f971006e2f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c15df10 0x7f972c0c59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f972c0b57c0 0x7f972c0c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 --2- 172.21.15.77:0/1789960767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f972c06b660 0x7f972c158300 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 -- 172.21.15.77:0/1789960767 >> 172.21.15.77:0/1789960767 conn(0x7f972c0d0a40 msgr2=0x7f972c0b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 -- 172.21.15.77:0/1789960767 shutdown_connections 2024-04-25T01:24:17.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.556+0000 7f973279e640 1 -- 172.21.15.77:0/1789960767 wait complete. 2024-04-25T01:24:17.548 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-25T01:24:17.186850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222600,"kb_used_data":6432,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752120,"statfs":{"total":773094113280,"available":772866170880,"internally_reserved":0,"allocated":6586368,"data_stored":3616920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17778,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-80,"num_read_kb":-94,"num_write":-66,"num_write_kb":-40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000830"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":235,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T01:24:01.954837+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:24:01.954837+0000","last_peered":"2024-04-25T01:24:01.954837+0000","last_clean":"2024-04-25T01:24:01.954837+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:24:01.954837+0000","last_undegraded":"2024-04-25T01:24:01.954837+0000","last_fullsized":"2024-04-25T01:24:01.954837+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1257472,"data_stored":886636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":303441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":655360,"data_stored":296144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":659456,"data_stored":296726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":692224,"data_stored":315594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":322175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":302991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1269760,"data_stored":893213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:24:17.549 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T01:24:17.549 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:24:17.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.664+0000 7f4292638640 1 Processor -- start 2024-04-25T01:24:17.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.664+0000 7f4292638640 1 -- start start 2024-04-25T01:24:17.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.665+0000 7f4292638640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f428c15e4c0 0x7f428c0d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.665+0000 7f4292638640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f428c0d1e90 con 0x7f428c0d4a10 2024-04-25T01:24:17.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.665+0000 7f4292638640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f428c0b54a0 con 0x7f428c0d2d40 2024-04-25T01:24:17.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.665+0000 7f4292638640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f428c0d2130 con 0x7f428c15e4c0 2024-04-25T01:24:17.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.665+0000 7f428b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f428c0d4a10 0x7f428c0b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38342/0 (socket says 172.21.15.77:38342) 2024-04-25T01:24:17.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.665+0000 7f428b7fe640 1 -- 172.21.15.77:0/4055992427 learned_addr learned my addr 172.21.15.77:0/4055992427 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:17.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.666+0000 7f428bfff640 1 --2- 172.21.15.77:0/4055992427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f428c15e4c0 0x7f428c0d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.666+0000 7f428affd640 1 -- 172.21.15.77:0/4055992427 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2648472292 0 0) 0x7f428c0d1e90 con 0x7f428c0d4a10 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.666+0000 7f428affd640 1 -- 172.21.15.77:0/4055992427 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4274003520 con 0x7f428c0d4a10 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.666+0000 7f428affd640 1 -- 172.21.15.77:0/4055992427 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3148663068 0 0) 0x7f428c0b54a0 con 0x7f428c0d2d40 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.666+0000 7f428affd640 1 -- 172.21.15.77:0/4055992427 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f428c0d1e90 con 0x7f428c0d2d40 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.666+0000 7f428affd640 1 -- 172.21.15.77:0/4055992427 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3837727123 0 0) 0x7f4274003520 con 0x7f428c0d4a10 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.666+0000 7f428bfff640 1 -- 172.21.15.77:0/4055992427 >> v1:172.21.15.121:6789/0 conn(0x7f428c0d2d40 legacy=0x7f428c0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.666+0000 7f428bfff640 1 -- 172.21.15.77:0/4055992427 >> v1:172.21.15.77:6789/0 conn(0x7f428c0d4a10 legacy=0x7f428c0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.666+0000 7f428bfff640 1 -- 172.21.15.77:0/4055992427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428c0b5780 con 0x7f428c15e4c0 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.667+0000 7f428bfff640 1 --2- 172.21.15.77:0/4055992427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f428c15e4c0 0x7f428c0d2800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f427c0097d0 tx=0x7f427c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a94280a1750e8dfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.667+0000 7f428affd640 1 -- 172.21.15.77:0/4055992427 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c037070 con 0x7f428c15e4c0 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.667+0000 7f428affd640 1 -- 172.21.15.77:0/4055992427 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427c02ab90 con 0x7f428c15e4c0 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.667+0000 7f428affd640 1 -- 172.21.15.77:0/4055992427 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4280001670 con 0x7f428c15e4c0 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.667+0000 7f4292638640 1 -- 172.21.15.77:0/4055992427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f428c15e4c0 msgr2=0x7f428c0d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.667+0000 7f4292638640 1 --2- 172.21.15.77:0/4055992427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f428c15e4c0 0x7f428c0d2800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f427c0097d0 tx=0x7f427c02a100 comp rx=0 tx=0).stop 2024-04-25T01:24:17.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.667+0000 7f4292638640 1 -- 172.21.15.77:0/4055992427 shutdown_connections 2024-04-25T01:24:17.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.667+0000 7f4292638640 1 --2- 172.21.15.77:0/4055992427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f428c15e4c0 0x7f428c0d2800 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.667+0000 7f4292638640 1 -- 172.21.15.77:0/4055992427 >> 172.21.15.77:0/4055992427 conn(0x7f428c0d3ef0 msgr2=0x7f428c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:17.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.667+0000 7f4292638640 1 -- 172.21.15.77:0/4055992427 shutdown_connections 2024-04-25T01:24:17.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.667+0000 7f4292638640 1 -- 172.21.15.77:0/4055992427 wait complete. 2024-04-25T01:24:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.668+0000 7f4292638640 1 Processor -- start 2024-04-25T01:24:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f4292638640 1 -- start start 2024-04-25T01:24:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f4292638640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c0d2d40 0x7f428c1522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f4292638640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f428c0d4a10 0x7f428c1f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f4292638640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f428c15e4c0 0x7f428c0b9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f4292638640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f428c15ea20 con 0x7f428c0d2d40 2024-04-25T01:24:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f4292638640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f428c15e8a0 con 0x7f428c15e4c0 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f4292638640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f428c0b45f0 con 0x7f428c0d4a10 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f428bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c0d2d40 0x7f428c1522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f428bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c0d2d40 0x7f428c1522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49618/0 (socket says 172.21.15.77:49618) 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f428bfff640 1 -- 172.21.15.77:0/718789700 learned_addr learned my addr 172.21.15.77:0/718789700 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f428b7fe640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f428c0d4a10 0x7f428c1f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.669+0000 7f4290bd5640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f428c15e4c0 0x7f428c0b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.670+0000 7f428bfff640 1 -- 172.21.15.77:0/718789700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f428c0d4a10 msgr2=0x7f428c1f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.670+0000 7f428bfff640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f428c0d4a10 0x7f428c1f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.670+0000 7f428bfff640 1 -- 172.21.15.77:0/718789700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f428c15e4c0 msgr2=0x7f428c0b9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.670+0000 7f428bfff640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f428c15e4c0 0x7f428c0b9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.670+0000 7f428bfff640 1 -- 172.21.15.77:0/718789700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428c0c8ab0 con 0x7f428c0d2d40 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.670+0000 7f428bfff640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c0d2d40 0x7f428c1522e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f427c0097d0 tx=0x7f427c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.670+0000 7f4288ff9640 1 -- 172.21.15.77:0/718789700 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c041030 con 0x7f428c0d2d40 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.670+0000 7f4288ff9640 1 -- 172.21.15.77:0/718789700 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427c03d420 con 0x7f428c0d2d40 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.670+0000 7f4288ff9640 1 -- 172.21.15.77:0/718789700 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c037040 con 0x7f428c0d2d40 2024-04-25T01:24:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.670+0000 7f4292638640 1 -- 172.21.15.77:0/718789700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428c0c8d90 con 0x7f428c0d2d40 2024-04-25T01:24:17.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.670+0000 7f4292638640 1 -- 172.21.15.77:0/718789700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f428c0c9350 con 0x7f428c0d2d40 2024-04-25T01:24:17.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.671+0000 7f4292638640 1 -- 172.21.15.77:0/718789700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f428c068fb0 con 0x7f428c0d2d40 2024-04-25T01:24:17.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.672+0000 7f4288ff9640 1 -- 172.21.15.77:0/718789700 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f427c044070 con 0x7f428c0d2d40 2024-04-25T01:24:17.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.672+0000 7f4288ff9640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f427806be30 0x7f427806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.672+0000 7f4288ff9640 1 -- 172.21.15.77:0/718789700 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f427c0ab020 con 0x7f428c0d2d40 2024-04-25T01:24:17.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.672+0000 7f428b7fe640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f427806be30 0x7f427806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.672+0000 7f428b7fe640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f427806be30 0x7f427806e2f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f428c0d4440 tx=0x7f4280001650 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.676+0000 7f4288ff9640 1 -- 172.21.15.77:0/718789700 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f427c0795f0 con 0x7f428c0d2d40 2024-04-25T01:24:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.846+0000 7f4292638640 1 -- 172.21.15.77:0/718789700 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f428c0d0f10 con 0x7f427806be30 2024-04-25T01:24:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.847+0000 7f4288ff9640 1 -- 172.21.15.77:0/718789700 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13419 (secure 0 0 0) 0x7f428c0d0f10 con 0x7f427806be30 2024-04-25T01:24:17.828 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:24:17.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 -- 172.21.15.77:0/718789700 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f427806be30 msgr2=0x7f427806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f427806be30 0x7f427806e2f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f428c0d4440 tx=0x7f4280001650 comp rx=0 tx=0).stop 2024-04-25T01:24:17.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 -- 172.21.15.77:0/718789700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c0d2d40 msgr2=0x7f428c1522e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c0d2d40 0x7f428c1522e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f427c0097d0 tx=0x7f427c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 -- 172.21.15.77:0/718789700 shutdown_connections 2024-04-25T01:24:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f427806be30 0x7f427806e2f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f428c15e4c0 0x7f428c0b9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f428c0d4a10 0x7f428c1f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 --2- 172.21.15.77:0/718789700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c0d2d40 0x7f428c1522e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 -- 172.21.15.77:0/718789700 >> 172.21.15.77:0/718789700 conn(0x7f428c0d3ef0 msgr2=0x7f428c0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 -- 172.21.15.77:0/718789700 shutdown_connections 2024-04-25T01:24:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.849+0000 7f4292638640 1 -- 172.21.15.77:0/718789700 wait complete. 2024-04-25T01:24:17.841 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-25T01:24:17.186850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222600,"kb_used_data":6432,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752120,"statfs":{"total":773094113280,"available":772866170880,"internally_reserved":0,"allocated":6586368,"data_stored":3616920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17778,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-80,"num_read_kb":-94,"num_write":-66,"num_write_kb":-40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000830"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":235,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T01:24:01.954837+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:24:01.954837+0000","last_peered":"2024-04-25T01:24:01.954837+0000","last_clean":"2024-04-25T01:24:01.954837+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:24:01.954837+0000","last_undegraded":"2024-04-25T01:24:01.954837+0000","last_fullsized":"2024-04-25T01:24:01.954837+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1257472,"data_stored":886636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":303441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":655360,"data_stored":296144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":659456,"data_stored":296726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":692224,"data_stored":315594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":322175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":302991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1269760,"data_stored":893213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:24:17.842 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:24:17.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.957+0000 7f396de3f640 1 Processor -- start 2024-04-25T01:24:17.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.957+0000 7f396de3f640 1 -- start start 2024-04-25T01:24:17.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f396de3f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f396815f9b0 0x7f396815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f396de3f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f396806d270 0x7f396806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f396de3f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39680b4240 con 0x7f3968145590 2024-04-25T01:24:17.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f396de3f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3968160830 con 0x7f396815f9b0 2024-04-25T01:24:17.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f396de3f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3968160350 con 0x7f396806d270 2024-04-25T01:24:17.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f3966ffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f396815f9b0 0x7f396815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f3967fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f396806d270 0x7f396806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f3966ffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f396815f9b0 0x7f396815fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.77:43222/0 (socket says 172.21.15.77:43222) 2024-04-25T01:24:17.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f3966ffd640 1 -- 172.21.15.77:0/4171746495 learned_addr learned my addr 172.21.15.77:0/4171746495 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:17.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f3966ffd640 1 -- 172.21.15.77:0/4171746495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f396806d270 msgr2=0x7f396806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:17.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f3966ffd640 1 --2- 172.21.15.77:0/4171746495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f396806d270 0x7f396806d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f3966ffd640 1 -- 172.21.15.77:0/4171746495 >> v1:172.21.15.121:6789/0 conn(0x7f3968145590 legacy=0x7f39680b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.958+0000 7f3966ffd640 1 -- 172.21.15.77:0/4171746495 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39680b39a0 con 0x7f396815f9b0 2024-04-25T01:24:17.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.959+0000 7f3966ffd640 1 --2- 172.21.15.77:0/4171746495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396815f9b0 0x7f396815fe10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f395c009870 tx=0x7f395c02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d623ca1a36c7a90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.959+0000 7f39667fc640 1 -- 172.21.15.77:0/4171746495 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c037070 con 0x7f396815f9b0 2024-04-25T01:24:17.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.959+0000 7f39667fc640 1 -- 172.21.15.77:0/4171746495 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f395c032440 con 0x7f396815f9b0 2024-04-25T01:24:17.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.959+0000 7f39667fc640 1 -- 172.21.15.77:0/4171746495 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c032810 con 0x7f396815f9b0 2024-04-25T01:24:17.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.960+0000 7f396de3f640 1 -- 172.21.15.77:0/4171746495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396815f9b0 msgr2=0x7f396815fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.960+0000 7f396de3f640 1 --2- 172.21.15.77:0/4171746495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396815f9b0 0x7f396815fe10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f395c009870 tx=0x7f395c02a270 comp rx=0 tx=0).stop 2024-04-25T01:24:17.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.960+0000 7f396de3f640 1 -- 172.21.15.77:0/4171746495 shutdown_connections 2024-04-25T01:24:17.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.960+0000 7f396de3f640 1 --2- 172.21.15.77:0/4171746495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f396806d270 0x7f396806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.960+0000 7f396de3f640 1 --2- 172.21.15.77:0/4171746495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396815f9b0 0x7f396815fe10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.960+0000 7f396de3f640 1 -- 172.21.15.77:0/4171746495 >> 172.21.15.77:0/4171746495 conn(0x7f39680b1820 msgr2=0x7f39680b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:17.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.960+0000 7f396de3f640 1 -- 172.21.15.77:0/4171746495 shutdown_connections 2024-04-25T01:24:17.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.960+0000 7f396de3f640 1 -- 172.21.15.77:0/4171746495 wait complete. 2024-04-25T01:24:17.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.961+0000 7f396de3f640 1 Processor -- start 2024-04-25T01:24:17.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.961+0000 7f396de3f640 1 -- start start 2024-04-25T01:24:17.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.962+0000 7f396de3f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396806d270 0x7f396813f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.962+0000 7f396de3f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3968145590 0x7f39681413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.962+0000 7f396de3f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f396815f9b0 0x7f3968141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.962+0000 7f396de3f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f39680b3f60 con 0x7f3968145590 2024-04-25T01:24:17.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.962+0000 7f396de3f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f396806cde0 con 0x7f396806d270 2024-04-25T01:24:17.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.962+0000 7f396de3f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3968145970 con 0x7f396815f9b0 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.962+0000 7f3966ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3968145590 0x7f39681413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.962+0000 7f39677fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396806d270 0x7f396813f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.962+0000 7f3966ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3968145590 0x7f39681413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49632/0 (socket says 172.21.15.77:49632) 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.962+0000 7f3966ffd640 1 -- 172.21.15.77:0/292907624 learned_addr learned my addr 172.21.15.77:0/292907624 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.962+0000 7f3967fff640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f396815f9b0 0x7f3968141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.963+0000 7f3966ffd640 1 -- 172.21.15.77:0/292907624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f396815f9b0 msgr2=0x7f3968141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.963+0000 7f3966ffd640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f396815f9b0 0x7f3968141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.963+0000 7f3966ffd640 1 -- 172.21.15.77:0/292907624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396806d270 msgr2=0x7f396813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.963+0000 7f3966ffd640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396806d270 0x7f396813f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.963+0000 7f3966ffd640 1 -- 172.21.15.77:0/292907624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3968143da0 con 0x7f3968145590 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.963+0000 7f3966ffd640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3968145590 0x7f39681413f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f395c02a2b0 tx=0x7f395c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.963+0000 7f394ffff640 1 -- 172.21.15.77:0/292907624 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c037040 con 0x7f3968145590 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.963+0000 7f394ffff640 1 -- 172.21.15.77:0/292907624 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f395c042070 con 0x7f3968145590 2024-04-25T01:24:17.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.963+0000 7f396de3f640 1 -- 172.21.15.77:0/292907624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3968144080 con 0x7f3968145590 2024-04-25T01:24:17.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.963+0000 7f394ffff640 1 -- 172.21.15.77:0/292907624 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c030880 con 0x7f3968145590 2024-04-25T01:24:17.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.963+0000 7f396de3f640 1 -- 172.21.15.77:0/292907624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3968144640 con 0x7f3968145590 2024-04-25T01:24:17.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.964+0000 7f396de3f640 1 -- 172.21.15.77:0/292907624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3968068fb0 con 0x7f3968145590 2024-04-25T01:24:17.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.964+0000 7f394ffff640 1 -- 172.21.15.77:0/292907624 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f395c030a20 con 0x7f3968145590 2024-04-25T01:24:17.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.965+0000 7f394ffff640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f395406be60 0x7f395406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.965+0000 7f394ffff640 1 -- 172.21.15.77:0/292907624 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f395c0ab4c0 con 0x7f3968145590 2024-04-25T01:24:17.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.965+0000 7f39677fe640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f395406be60 0x7f395406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.965+0000 7f39677fe640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f395406be60 0x7f395406e320 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3958001780 tx=0x7f3958009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:17.969+0000 7f394ffff640 1 -- 172.21.15.77:0/292907624 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f395c0798e0 con 0x7f3968145590 2024-04-25T01:24:18.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.143+0000 7f396de3f640 1 -- 172.21.15.77:0/292907624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f396806ea50 con 0x7f3968145590 2024-04-25T01:24:18.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.144+0000 7f394ffff640 1 -- 172.21.15.77:0/292907624 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+13555 (secure 0 0 0) 0x7f395c07dbe0 con 0x7f3968145590 2024-04-25T01:24:18.125 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:18.125 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"fsid":"7c3c39ee-077e-476c-8994-eede4fab301d","created":"2024-04-25T01:21:21.098696+0000","modified":"2024-04-25T01:24:15.029152+0000","last_up_change":"2024-04-25T01:21:32.237749+0000","last_in_change":"2024-04-25T01:21:23.312611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:21:33.186152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40e3ffde-0d89-45b6-acf6-b17d1c5a5114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6825","nonce":4267986699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6827","nonce":4267986699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6831","nonce":4267986699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":4267986699},{"type":"v1","addr":"172.21.15.77:6829","nonce":4267986699}]},"public_addr":"172.21.15.77:6825/4267986699","cluster_addr":"172.21.15.77:6827/4267986699","heartbeat_back_addr":"172.21.15.77:6831/4267986699","heartbeat_front_addr":"172.21.15.77:6829/4267986699","state":["exists","up"]},{"osd":1,"uuid":"3054f2ce-6486-4603-bc78-52bcefbd2aef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6811","nonce":3460510388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6813","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6815","nonce":3460510388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6822","nonce":3460510388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":3460510388},{"type":"v1","addr":"172.21.15.77:6818","nonce":3460510388}]},"public_addr":"172.21.15.77:6811/3460510388","cluster_addr":"172.21.15.77:6815/3460510388","heartbeat_back_addr":"172.21.15.77:6822/3460510388","heartbeat_front_addr":"172.21.15.77:6818/3460510388","state":["exists","up"]},{"osd":2,"uuid":"7f461099-df7e-4ae9-bd92-689e529ea919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6801","nonce":1130238186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6803","nonce":1130238186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6810","nonce":1130238186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1130238186},{"type":"v1","addr":"172.21.15.77:6805","nonce":1130238186}]},"public_addr":"172.21.15.77:6801/1130238186","cluster_addr":"172.21.15.77:6803/1130238186","heartbeat_back_addr":"172.21.15.77:6810/1130238186","heartbeat_front_addr":"172.21.15.77:6805/1130238186","state":["exists","up"]},{"osd":3,"uuid":"1c3b7b96-f9b6-461c-a541-cf96843238d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6809","nonce":3656221607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6814","nonce":3656221607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6821","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6823","nonce":3656221607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3656221607},{"type":"v1","addr":"172.21.15.77:6819","nonce":3656221607}]},"public_addr":"172.21.15.77:6809/3656221607","cluster_addr":"172.21.15.77:6814/3656221607","heartbeat_back_addr":"172.21.15.77:6823/3656221607","heartbeat_front_addr":"172.21.15.77:6819/3656221607","state":["exists","up"]},{"osd":4,"uuid":"17a4c982-a80b-4b7d-8964-b8b476bca624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6803","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6805","nonce":3650663429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6809","nonce":3650663429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6815","nonce":3650663429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":3650663429},{"type":"v1","addr":"172.21.15.121:6813","nonce":3650663429}]},"public_addr":"172.21.15.121:6805/3650663429","cluster_addr":"172.21.15.121:6809/3650663429","heartbeat_back_addr":"172.21.15.121:6815/3650663429","heartbeat_front_addr":"172.21.15.121:6813/3650663429","state":["exists","up"]},{"osd":5,"uuid":"13fc3195-286c-4f2d-9d9d-e48d1adb9294","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6819","nonce":3849358101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6823","nonce":3849358101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6829","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6831","nonce":3849358101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6825","nonce":3849358101},{"type":"v1","addr":"172.21.15.121:6827","nonce":3849358101}]},"public_addr":"172.21.15.121:6819/3849358101","cluster_addr":"172.21.15.121:6823/3849358101","heartbeat_back_addr":"172.21.15.121:6831/3849358101","heartbeat_front_addr":"172.21.15.121:6827/3849358101","state":["exists","up"]},{"osd":6,"uuid":"7c2ee0f6-6136-4fde-89a2-5be0ba7879b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6818","nonce":3890411649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6822","nonce":3890411649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6830","nonce":3890411649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3890411649},{"type":"v1","addr":"172.21.15.121:6826","nonce":3890411649}]},"public_addr":"172.21.15.121:6818/3890411649","cluster_addr":"172.21.15.121:6822/3890411649","heartbeat_back_addr":"172.21.15.121:6830/3890411649","heartbeat_front_addr":"172.21.15.121:6826/3890411649","state":["exists","up"]},{"osd":7,"uuid":"79293f60-29fe-4279-90a3-5d95cd553b00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6801","nonce":251765690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6804","nonce":251765690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6812","nonce":251765690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":251765690},{"type":"v1","addr":"172.21.15.121:6808","nonce":251765690}]},"public_addr":"172.21.15.121:6801/251765690","cluster_addr":"172.21.15.121:6804/251765690","heartbeat_back_addr":"172.21.15.121:6812/251765690","heartbeat_front_addr":"172.21.15.121:6808/251765690","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.525464+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.457393+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.405538+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.504734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.482195+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.508935+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.449989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:28.524520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6835/2057169374":"2024-04-26T01:24:11.999175+0000","172.21.15.77:6834/2057169374":"2024-04-26T01:24:11.999175+0000","172.21.15.77:6836/3189776231":"2024-04-26T01:24:01.947151+0000","172.21.15.77:6834/3189776231":"2024-04-26T01:24:01.947151+0000","172.21.15.121:6834/3834135103":"2024-04-26T01:24:00.948815+0000","172.21.15.121:6832/3834135103":"2024-04-26T01:24:00.948815+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 -- 172.21.15.77:0/292907624 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f395406be60 msgr2=0x7f395406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f395406be60 0x7f395406e320 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3958001780 tx=0x7f3958009040 comp rx=0 tx=0).stop 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 -- 172.21.15.77:0/292907624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3968145590 msgr2=0x7f39681413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3968145590 0x7f39681413f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f395c02a2b0 tx=0x7f395c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 -- 172.21.15.77:0/292907624 shutdown_connections 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f395406be60 0x7f395406e320 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3968145590 0x7f39681413f0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f396815f9b0 0x7f3968141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 --2- 172.21.15.77:0/292907624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396806d270 0x7f396813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 -- 172.21.15.77:0/292907624 >> 172.21.15.77:0/292907624 conn(0x7f39680b1820 msgr2=0x7f39680b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 -- 172.21.15.77:0/292907624 shutdown_connections 2024-04-25T01:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:18.146+0000 7f396de3f640 1 -- 172.21.15.77:0/292907624 wait complete. 2024-04-25T01:24:19.139 INFO:tasks.ceph:Scrubbing osd.0 2024-04-25T01:24:19.140 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:24:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.261+0000 7f635084d640 1 Processor -- start 2024-04-25T01:24:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.261+0000 7f635084d640 1 -- start start 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.262+0000 7f635084d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634c0ce470 0x7f634c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.262+0000 7f635084d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f634c06e830 0x7f634c071470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.262+0000 7f635084d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f634c0cf700 con 0x7f634c15ae90 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.262+0000 7f635084d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f634c0cd6b0 con 0x7f634c06e830 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.262+0000 7f635084d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f634c0cd990 con 0x7f634c0ce470 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.262+0000 7f634ad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f634c06e830 0x7f634c071470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.262+0000 7f634a59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f634c15ae90 0x7f634c0ceb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38348/0 (socket says 172.21.15.77:38348) 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.262+0000 7f634a59c640 1 -- 172.21.15.77:0/1177367784 learned_addr learned my addr 172.21.15.77:0/1177367784 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.262+0000 7f6343fff640 1 --2- 172.21.15.77:0/1177367784 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634c0ce470 0x7f634c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.263+0000 7f634ad9d640 1 -- 172.21.15.77:0/1177367784 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634c0ce470 msgr2=0x7f634c06e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.263+0000 7f634ad9d640 1 --2- 172.21.15.77:0/1177367784 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634c0ce470 0x7f634c06e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.263+0000 7f634ad9d640 1 -- 172.21.15.77:0/1177367784 >> v1:172.21.15.77:6789/0 conn(0x7f634c15ae90 legacy=0x7f634c0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.263+0000 7f634ad9d640 1 -- 172.21.15.77:0/1177367784 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634c0cdc70 con 0x7f634c06e830 2024-04-25T01:24:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.263+0000 7f6349d9b640 1 -- 172.21.15.77:0/1177367784 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3144946162 0 0) 0x7f634c0cf700 con 0x7f634c15ae90 2024-04-25T01:24:19.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.263+0000 7f634ad9d640 1 --2- 172.21.15.77:0/1177367784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f634c06e830 0x7f634c071470 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6344009d90 tx=0x7f634402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9def13c7fe3d7d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.264+0000 7f6349d9b640 1 -- 172.21.15.77:0/1177367784 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344007740 con 0x7f634c06e830 2024-04-25T01:24:19.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.264+0000 7f6349d9b640 1 -- 172.21.15.77:0/1177367784 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63440078e0 con 0x7f634c06e830 2024-04-25T01:24:19.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.264+0000 7f6349d9b640 1 -- 172.21.15.77:0/1177367784 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344007cb0 con 0x7f634c06e830 2024-04-25T01:24:19.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.264+0000 7f635084d640 1 -- 172.21.15.77:0/1177367784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f634c06e830 msgr2=0x7f634c071470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.264+0000 7f635084d640 1 --2- 172.21.15.77:0/1177367784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f634c06e830 0x7f634c071470 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6344009d90 tx=0x7f634402a660 comp rx=0 tx=0).stop 2024-04-25T01:24:19.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.264+0000 7f635084d640 1 -- 172.21.15.77:0/1177367784 shutdown_connections 2024-04-25T01:24:19.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.264+0000 7f635084d640 1 --2- 172.21.15.77:0/1177367784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f634c06e830 0x7f634c071470 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.264+0000 7f635084d640 1 --2- 172.21.15.77:0/1177367784 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634c0ce470 0x7f634c06e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.264+0000 7f635084d640 1 -- 172.21.15.77:0/1177367784 >> 172.21.15.77:0/1177367784 conn(0x7f634c0cfae0 msgr2=0x7f634c0cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:19.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.264+0000 7f635084d640 1 -- 172.21.15.77:0/1177367784 shutdown_connections 2024-04-25T01:24:19.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.264+0000 7f635084d640 1 -- 172.21.15.77:0/1177367784 wait complete. 2024-04-25T01:24:19.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.265+0000 7f635084d640 1 Processor -- start 2024-04-25T01:24:19.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f635084d640 1 -- start start 2024-04-25T01:24:19.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f635084d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f634c06e830 0x7f634c0c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f635084d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634c0ce470 0x7f634c14d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f635084d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f634c15ae90 0x7f634c14dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f635084d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f634c0cf2a0 con 0x7f634c06e830 2024-04-25T01:24:19.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f635084d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f634c06cf40 con 0x7f634c15ae90 2024-04-25T01:24:19.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f635084d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f634c0cf420 con 0x7f634c0ce470 2024-04-25T01:24:19.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f634a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f634c06e830 0x7f634c0c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f634a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f634c06e830 0x7f634c0c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49654/0 (socket says 172.21.15.77:49654) 2024-04-25T01:24:19.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f634a59c640 1 -- 172.21.15.77:0/939078407 learned_addr learned my addr 172.21.15.77:0/939078407 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f6343fff640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634c0ce470 0x7f634c14d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.266+0000 7f634ad9d640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f634c15ae90 0x7f634c14dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.267+0000 7f634a59c640 1 -- 172.21.15.77:0/939078407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634c0ce470 msgr2=0x7f634c14d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.267+0000 7f634a59c640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634c0ce470 0x7f634c14d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.267+0000 7f634a59c640 1 -- 172.21.15.77:0/939078407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f634c15ae90 msgr2=0x7f634c14dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.267+0000 7f634a59c640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f634c15ae90 0x7f634c14dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.267+0000 7f634a59c640 1 -- 172.21.15.77:0/939078407 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634c0b5590 con 0x7f634c06e830 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.267+0000 7f634a59c640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f634c06e830 0x7f634c0c90a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f633c00c430 tx=0x7f633c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.267+0000 7f6342ffd640 1 -- 172.21.15.77:0/939078407 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c0074e0 con 0x7f634c06e830 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.267+0000 7f6342ffd640 1 -- 172.21.15.77:0/939078407 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f633c0047e0 con 0x7f634c06e830 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.267+0000 7f635084d640 1 -- 172.21.15.77:0/939078407 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f634c0b5870 con 0x7f634c06e830 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.267+0000 7f6342ffd640 1 -- 172.21.15.77:0/939078407 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c004bd0 con 0x7f634c06e830 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.267+0000 7f635084d640 1 -- 172.21.15.77:0/939078407 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f634c15c300 con 0x7f634c06e830 2024-04-25T01:24:19.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.268+0000 7f635084d640 1 -- 172.21.15.77:0/939078407 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f634c068fb0 con 0x7f634c06e830 2024-04-25T01:24:19.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.268+0000 7f6342ffd640 1 -- 172.21.15.77:0/939078407 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f633c011070 con 0x7f634c06e830 2024-04-25T01:24:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.269+0000 7f6342ffd640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f632c06be60 0x7f632c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.269+0000 7f6342ffd640 1 -- 172.21.15.77:0/939078407 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f633c08a8c0 con 0x7f634c06e830 2024-04-25T01:24:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.269+0000 7f6343fff640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f632c06be60 0x7f632c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.269+0000 7f6342ffd640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] conn(0x7f632c072440 0x7f632c074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.269+0000 7f6342ffd640 1 -- 172.21.15.77:0/939078407 --> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f633c01ce80 con 0x7f632c072440 2024-04-25T01:24:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.269+0000 7f6342ffd640 1 -- 172.21.15.77:0/939078407 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f633c08f2c0 con 0x7f634c06e830 2024-04-25T01:24:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.269+0000 7f634ad9d640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] conn(0x7f632c072440 0x7f632c074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.269+0000 7f6343fff640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f632c06be60 0x7f632c06e320 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f634c0d04f0 tx=0x7f6334009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.269+0000 7f634ad9d640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] conn(0x7f632c072440 0x7f632c074880 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.270+0000 7f6342ffd640 1 -- 172.21.15.77:0/939078407 <== osd.0 v2:172.21.15.77:6824/4267986699 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f633c01ce80 con 0x7f632c072440 2024-04-25T01:24:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.283+0000 7f635084d640 1 -- 172.21.15.77:0/939078407 --> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f634c15f5d0 con 0x7f632c072440 2024-04-25T01:24:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.283+0000 7f6342ffd640 1 -- 172.21.15.77:0/939078407 <== osd.0 v2:172.21.15.77:6824/4267986699 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f634c15f5d0 con 0x7f632c072440 2024-04-25T01:24:19.264 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:24:19.264 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:24:19.264 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:24:19.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.283+0000 7f635084d640 1 -- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] conn(0x7f632c072440 msgr2=0x7f632c074880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.283+0000 7f635084d640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:6824/4267986699,v1:172.21.15.77:6825/4267986699] conn(0x7f632c072440 0x7f632c074880 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.284+0000 7f635084d640 1 -- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f632c06be60 msgr2=0x7f632c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.284+0000 7f635084d640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f632c06be60 0x7f632c06e320 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f634c0d04f0 tx=0x7f6334009190 comp rx=0 tx=0).stop 2024-04-25T01:24:19.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.284+0000 7f635084d640 1 -- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f634c06e830 msgr2=0x7f634c0c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.284+0000 7f635084d640 1 --2- 172.21.15.77:0/939078407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f634c06e830 0x7f634c0c90a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f633c00c430 tx=0x7f633c00c8f0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.284+0000 7f634ad9d640 1 -- 172.21.15.77:0/939078407 reap_dead start 2024-04-25T01:24:19.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.284+0000 7f635084d640 1 -- 172.21.15.77:0/939078407 shutdown_connections 2024-04-25T01:24:19.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.284+0000 7f635084d640 1 -- 172.21.15.77:0/939078407 >> 172.21.15.77:0/939078407 conn(0x7f634c0cfae0 msgr2=0x7f634c157930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:19.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.284+0000 7f635084d640 1 -- 172.21.15.77:0/939078407 shutdown_connections 2024-04-25T01:24:19.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.284+0000 7f635084d640 1 -- 172.21.15.77:0/939078407 wait complete. 2024-04-25T01:24:19.274 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-25T01:24:19.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.401+0000 7f49b689a640 1 Processor -- start 2024-04-25T01:24:19.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.402+0000 7f49b689a640 1 -- start start 2024-04-25T01:24:19.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.402+0000 7f49b689a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f49b015e4c0 0x7f49b00d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.402+0000 7f49b689a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49b00d4a10 0x7f49b00b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.402+0000 7f49b689a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49b00d1e90 con 0x7f49b00d2d40 2024-04-25T01:24:19.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.402+0000 7f49b689a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f49b00b54a0 con 0x7f49b015e4c0 2024-04-25T01:24:19.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.402+0000 7f49b689a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f49b00d2130 con 0x7f49b00d4a10 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.402+0000 7f49affff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f49b015e4c0 0x7f49b00d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.402+0000 7f49affff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f49b015e4c0 0x7f49b00d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49658/0 (socket says 172.21.15.77:49658) 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.402+0000 7f49affff640 1 -- 172.21.15.77:0/1483980885 learned_addr learned my addr 172.21.15.77:0/1483980885 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.402+0000 7f49b4e37640 1 --1- 172.21.15.77:0/1483980885 >> v1:172.21.15.121:6789/0 conn(0x7f49b00d2d40 0x7f49b00d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.77:45808/0 (socket says 172.21.15.77:45808) 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.402+0000 7f49af7fe640 1 --2- 172.21.15.77:0/1483980885 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49b00d4a10 0x7f49b00b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.403+0000 7f49affff640 1 -- 172.21.15.77:0/1483980885 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49b00d4a10 msgr2=0x7f49b00b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.403+0000 7f49affff640 1 --2- 172.21.15.77:0/1483980885 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49b00d4a10 0x7f49b00b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.403+0000 7f49affff640 1 -- 172.21.15.77:0/1483980885 >> v1:172.21.15.121:6789/0 conn(0x7f49b00d2d40 legacy=0x7f49b00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.403+0000 7f49aeffd640 1 -- 172.21.15.77:0/1483980885 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2864941706 0 0) 0x7f49b00d1e90 con 0x7f49b00d2d40 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.403+0000 7f49affff640 1 -- 172.21.15.77:0/1483980885 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b00b5780 con 0x7f49b015e4c0 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.403+0000 7f49af7fe640 1 --2- 172.21.15.77:0/1483980885 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49b00d4a10 0x7f49b00b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.403+0000 7f49affff640 1 --2- 172.21.15.77:0/1483980885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b015e4c0 0x7f49b00d2800 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009870 tx=0x7f49a002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b7aad18f82e7b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.404+0000 7f49aeffd640 1 -- 172.21.15.77:0/1483980885 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0037070 con 0x7f49b015e4c0 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.404+0000 7f49aeffd640 1 -- 172.21.15.77:0/1483980885 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49a0032440 con 0x7f49b015e4c0 2024-04-25T01:24:19.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.404+0000 7f49aeffd640 1 -- 172.21.15.77:0/1483980885 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0032810 con 0x7f49b015e4c0 2024-04-25T01:24:19.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.404+0000 7f49b689a640 1 -- 172.21.15.77:0/1483980885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b015e4c0 msgr2=0x7f49b00d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.404+0000 7f49b689a640 1 --2- 172.21.15.77:0/1483980885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b015e4c0 0x7f49b00d2800 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009870 tx=0x7f49a002a1c0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.404+0000 7f49b689a640 1 -- 172.21.15.77:0/1483980885 shutdown_connections 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.404+0000 7f49b689a640 1 --2- 172.21.15.77:0/1483980885 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49b00d4a10 0x7f49b00b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.404+0000 7f49b689a640 1 --2- 172.21.15.77:0/1483980885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b015e4c0 0x7f49b00d2800 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.404+0000 7f49b689a640 1 -- 172.21.15.77:0/1483980885 >> 172.21.15.77:0/1483980885 conn(0x7f49b00d3ef0 msgr2=0x7f49b00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.404+0000 7f49b689a640 1 -- 172.21.15.77:0/1483980885 shutdown_connections 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.404+0000 7f49b689a640 1 -- 172.21.15.77:0/1483980885 wait complete. 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.405+0000 7f49b689a640 1 Processor -- start 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49b689a640 1 -- start start 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49b689a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b00d2d40 0x7f49b01578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49b689a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49b00d4a10 0x7f49b00c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49b689a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49b015e4c0 0x7f49b00c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49b689a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49b015ea20 con 0x7f49b00d2d40 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49b689a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f49b015e8a0 con 0x7f49b00d4a10 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49b689a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f49b00d1010 con 0x7f49b015e4c0 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b00d2d40 0x7f49b01578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b00d2d40 0x7f49b01578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49672/0 (socket says 172.21.15.77:49672) 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49affff640 1 -- 172.21.15.77:0/1019796153 learned_addr learned my addr 172.21.15.77:0/1019796153 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49b4e37640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49b015e4c0 0x7f49b00c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.406+0000 7f49af7fe640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49b00d4a10 0x7f49b00c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.407+0000 7f49affff640 1 -- 172.21.15.77:0/1019796153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49b015e4c0 msgr2=0x7f49b00c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.407+0000 7f49affff640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49b015e4c0 0x7f49b00c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.407+0000 7f49affff640 1 -- 172.21.15.77:0/1019796153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49b00d4a10 msgr2=0x7f49b00c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.407+0000 7f49affff640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49b00d4a10 0x7f49b00c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.407+0000 7f49affff640 1 -- 172.21.15.77:0/1019796153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b0158410 con 0x7f49b00d2d40 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.407+0000 7f49affff640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b00d2d40 0x7f49b01578f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009870 tx=0x7f49a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.407+0000 7f49acff9640 1 -- 172.21.15.77:0/1019796153 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0041070 con 0x7f49b00d2d40 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.407+0000 7f49acff9640 1 -- 172.21.15.77:0/1019796153 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49a003c890 con 0x7f49b00d2d40 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.407+0000 7f49b689a640 1 -- 172.21.15.77:0/1019796153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b00b9c50 con 0x7f49b00d2d40 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.407+0000 7f49acff9640 1 -- 172.21.15.77:0/1019796153 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0037040 con 0x7f49b00d2d40 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.407+0000 7f49b689a640 1 -- 172.21.15.77:0/1019796153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b00b7700 con 0x7f49b00d2d40 2024-04-25T01:24:19.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.408+0000 7f49b689a640 1 -- 172.21.15.77:0/1019796153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b01585e0 con 0x7f49b00d2d40 2024-04-25T01:24:19.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.408+0000 7f49acff9640 1 -- 172.21.15.77:0/1019796153 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f49a0044070 con 0x7f49b00d2d40 2024-04-25T01:24:19.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.409+0000 7f49acff9640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f499c06be30 0x7f499c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.409+0000 7f49acff9640 1 -- 172.21.15.77:0/1019796153 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f49a00ab830 con 0x7f49b00d2d40 2024-04-25T01:24:19.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.409+0000 7f49af7fe640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f499c06be30 0x7f499c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.409+0000 7f49af7fe640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f499c06be30 0x7f499c06e2f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009560 tx=0x7f49a4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.413+0000 7f49acff9640 1 -- 172.21.15.77:0/1019796153 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f49a0079e00 con 0x7f49b00d2d40 2024-04-25T01:24:19.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.586+0000 7f49b689a640 1 -- 172.21.15.77:0/1019796153 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f49b00b9a00 con 0x7f499c06be30 2024-04-25T01:24:19.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.586+0000 7f49acff9640 1 -- 172.21.15.77:0/1019796153 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f49b00b9a00 con 0x7f499c06be30 2024-04-25T01:24:19.567 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T01:24:19.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 -- 172.21.15.77:0/1019796153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f499c06be30 msgr2=0x7f499c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f499c06be30 0x7f499c06e2f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009560 tx=0x7f49a4009190 comp rx=0 tx=0).stop 2024-04-25T01:24:19.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 -- 172.21.15.77:0/1019796153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b00d2d40 msgr2=0x7f49b01578f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b00d2d40 0x7f49b01578f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009870 tx=0x7f49a00046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 -- 172.21.15.77:0/1019796153 shutdown_connections 2024-04-25T01:24:19.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f499c06be30 0x7f499c06e2f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49b015e4c0 0x7f49b00c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49b00d4a10 0x7f49b00c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 --2- 172.21.15.77:0/1019796153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b00d2d40 0x7f49b01578f0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 -- 172.21.15.77:0/1019796153 >> 172.21.15.77:0/1019796153 conn(0x7f49b00d3ef0 msgr2=0x7f49b00d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:19.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 -- 172.21.15.77:0/1019796153 shutdown_connections 2024-04-25T01:24:19.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.588+0000 7f49b689a640 1 -- 172.21.15.77:0/1019796153 wait complete. 2024-04-25T01:24:19.580 INFO:tasks.ceph:Scrubbing osd.1 2024-04-25T01:24:19.580 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:24:19.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.700+0000 7ff790e4d640 1 Processor -- start 2024-04-25T01:24:19.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.701+0000 7ff790e4d640 1 -- start start 2024-04-25T01:24:19.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.701+0000 7ff790e4d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff78c15e4c0 0x7ff78c0d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.701+0000 7ff790e4d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff78c0d2d40 0x7ff78c0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.701+0000 7ff790e4d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff78c0d1e90 con 0x7ff78c0d4a10 2024-04-25T01:24:19.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.701+0000 7ff790e4d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7ff78c0b54a0 con 0x7ff78c15e4c0 2024-04-25T01:24:19.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.701+0000 7ff790e4d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff78c0d2130 con 0x7ff78c0d2d40 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.701+0000 7ff78a59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff78c15e4c0 0x7ff78c0d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.702+0000 7ff78a59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff78c15e4c0 0x7ff78c0d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49688/0 (socket says 172.21.15.77:49688) 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.702+0000 7ff78a59c640 1 -- 172.21.15.77:0/2477658921 learned_addr learned my addr 172.21.15.77:0/2477658921 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.702+0000 7ff78ad9d640 1 --2- 172.21.15.77:0/2477658921 >> v2:172.21.15.121:3300/0 conn(0x7ff78c0d2d40 0x7ff78c0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.702+0000 7ff78a59c640 1 -- 172.21.15.77:0/2477658921 >> v2:172.21.15.121:3300/0 conn(0x7ff78c0d2d40 msgr2=0x7ff78c0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.702+0000 7ff78a59c640 1 --2- 172.21.15.77:0/2477658921 >> v2:172.21.15.121:3300/0 conn(0x7ff78c0d2d40 0x7ff78c0cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.702+0000 7ff78a59c640 1 -- 172.21.15.77:0/2477658921 >> v1:172.21.15.121:6789/0 conn(0x7ff78c0d4a10 legacy=0x7ff78c0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.702+0000 7ff78a59c640 1 -- 172.21.15.77:0/2477658921 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78c0b5780 con 0x7ff78c15e4c0 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.702+0000 7ff78a59c640 1 --2- 172.21.15.77:0/2477658921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff78c15e4c0 0x7ff78c0d2800 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff78c08d2e0 tx=0x7ff77c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11d5b36f82ce9e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.703+0000 7ff789d9b640 1 -- 172.21.15.77:0/2477658921 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c037070 con 0x7ff78c15e4c0 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.703+0000 7ff789d9b640 1 -- 172.21.15.77:0/2477658921 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77c032440 con 0x7ff78c15e4c0 2024-04-25T01:24:19.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.703+0000 7ff789d9b640 1 -- 172.21.15.77:0/2477658921 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c032810 con 0x7ff78c15e4c0 2024-04-25T01:24:19.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.703+0000 7ff790e4d640 1 -- 172.21.15.77:0/2477658921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff78c15e4c0 msgr2=0x7ff78c0d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.703+0000 7ff790e4d640 1 --2- 172.21.15.77:0/2477658921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff78c15e4c0 0x7ff78c0d2800 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff78c08d2e0 tx=0x7ff77c02a1e0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.703+0000 7ff790e4d640 1 -- 172.21.15.77:0/2477658921 shutdown_connections 2024-04-25T01:24:19.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.703+0000 7ff790e4d640 1 --2- 172.21.15.77:0/2477658921 >> v2:172.21.15.121:3300/0 conn(0x7ff78c0d2d40 0x7ff78c0cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.703+0000 7ff790e4d640 1 --2- 172.21.15.77:0/2477658921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff78c15e4c0 0x7ff78c0d2800 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.703+0000 7ff790e4d640 1 -- 172.21.15.77:0/2477658921 >> 172.21.15.77:0/2477658921 conn(0x7ff78c0d3ef0 msgr2=0x7ff78c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.703+0000 7ff790e4d640 1 -- 172.21.15.77:0/2477658921 shutdown_connections 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.703+0000 7ff790e4d640 1 -- 172.21.15.77:0/2477658921 wait complete. 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff790e4d640 1 Processor -- start 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff790e4d640 1 -- start start 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff790e4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff78c0d2d40 0x7ff78c150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff790e4d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff78c0d4a10 0x7ff78c0ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff790e4d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff78c15e4c0 0x7ff78c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff790e4d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c15ea20 con 0x7ff78c0d2d40 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff790e4d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c15e8a0 con 0x7ff78c0d4a10 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff790e4d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff78c0b45f0 con 0x7ff78c15e4c0 2024-04-25T01:24:19.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff78a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff78c0d2d40 0x7ff78c150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff78a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff78c0d2d40 0x7ff78c150220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49698/0 (socket says 172.21.15.77:49698) 2024-04-25T01:24:19.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff78a59c640 1 -- 172.21.15.77:0/2712515917 learned_addr learned my addr 172.21.15.77:0/2712515917 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:19.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff783fff640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff78c0d4a10 0x7ff78c0ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.705+0000 7ff78ad9d640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff78c15e4c0 0x7ff78c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff78a59c640 1 -- 172.21.15.77:0/2712515917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff78c15e4c0 msgr2=0x7ff78c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff78a59c640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff78c15e4c0 0x7ff78c15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff78a59c640 1 -- 172.21.15.77:0/2712515917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff78c0d4a10 msgr2=0x7ff78c0ca200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff78a59c640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff78c0d4a10 0x7ff78c0ca200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff78a59c640 1 -- 172.21.15.77:0/2712515917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78c1f7510 con 0x7ff78c0d2d40 2024-04-25T01:24:19.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff78ad9d640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff78c15e4c0 0x7ff78c15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:19.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff78a59c640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff78c0d2d40 0x7ff78c150220 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff77c02a220 tx=0x7ff77c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff782ffd640 1 -- 172.21.15.77:0/2712515917 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c037040 con 0x7ff78c0d2d40 2024-04-25T01:24:19.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff782ffd640 1 -- 172.21.15.77:0/2712515917 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77c042070 con 0x7ff78c0d2d40 2024-04-25T01:24:19.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff790e4d640 1 -- 172.21.15.77:0/2712515917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78c1f77b0 con 0x7ff78c0d2d40 2024-04-25T01:24:19.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff782ffd640 1 -- 172.21.15.77:0/2712515917 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c030880 con 0x7ff78c0d2d40 2024-04-25T01:24:19.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.706+0000 7ff790e4d640 1 -- 172.21.15.77:0/2712515917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff78c1f7d70 con 0x7ff78c0d2d40 2024-04-25T01:24:19.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.707+0000 7ff790e4d640 1 -- 172.21.15.77:0/2712515917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff78c068fb0 con 0x7ff78c0d2d40 2024-04-25T01:24:19.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.707+0000 7ff782ffd640 1 -- 172.21.15.77:0/2712515917 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff77c030a20 con 0x7ff78c0d2d40 2024-04-25T01:24:19.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.708+0000 7ff782ffd640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff76c06be30 0x7ff76c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.708+0000 7ff782ffd640 1 -- 172.21.15.77:0/2712515917 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7ff77c0ab610 con 0x7ff78c0d2d40 2024-04-25T01:24:19.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.708+0000 7ff783fff640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff76c06be30 0x7ff76c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.708+0000 7ff782ffd640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] conn(0x7ff76c072410 0x7ff76c074850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.708+0000 7ff782ffd640 1 -- 172.21.15.77:0/2712515917 --> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff77c031ec0 con 0x7ff76c072410 2024-04-25T01:24:19.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.708+0000 7ff782ffd640 1 -- 172.21.15.77:0/2712515917 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7ff77c0b2050 con 0x7ff78c0d2d40 2024-04-25T01:24:19.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.708+0000 7ff78ad9d640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] conn(0x7ff76c072410 0x7ff76c074850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.708+0000 7ff783fff640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff76c06be30 0x7ff76c06e2f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff78c0ce410 tx=0x7ff774009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.708+0000 7ff78ad9d640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] conn(0x7ff76c072410 0x7ff76c074850 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.709+0000 7ff782ffd640 1 -- 172.21.15.77:0/2712515917 <== osd.1 v2:172.21.15.77:6808/3460510388 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff77c031ec0 con 0x7ff76c072410 2024-04-25T01:24:19.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.722+0000 7ff790e4d640 1 -- 172.21.15.77:0/2712515917 --> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff78c06d8a0 con 0x7ff76c072410 2024-04-25T01:24:19.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.722+0000 7ff782ffd640 1 -- 172.21.15.77:0/2712515917 <== osd.1 v2:172.21.15.77:6808/3460510388 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff78c06d8a0 con 0x7ff76c072410 2024-04-25T01:24:19.703 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:24:19.703 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:24:19.703 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:24:19.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.723+0000 7ff790e4d640 1 -- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] conn(0x7ff76c072410 msgr2=0x7ff76c074850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.723+0000 7ff790e4d640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:6808/3460510388,v1:172.21.15.77:6811/3460510388] conn(0x7ff76c072410 0x7ff76c074850 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.723+0000 7ff790e4d640 1 -- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff76c06be30 msgr2=0x7ff76c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.723+0000 7ff790e4d640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff76c06be30 0x7ff76c06e2f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff78c0ce410 tx=0x7ff774009040 comp rx=0 tx=0).stop 2024-04-25T01:24:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.723+0000 7ff790e4d640 1 -- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff78c0d2d40 msgr2=0x7ff78c150220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.723+0000 7ff790e4d640 1 --2- 172.21.15.77:0/2712515917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff78c0d2d40 0x7ff78c150220 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff77c02a220 tx=0x7ff77c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.723+0000 7ff78ad9d640 1 -- 172.21.15.77:0/2712515917 reap_dead start 2024-04-25T01:24:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.723+0000 7ff790e4d640 1 -- 172.21.15.77:0/2712515917 shutdown_connections 2024-04-25T01:24:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.723+0000 7ff790e4d640 1 -- 172.21.15.77:0/2712515917 >> 172.21.15.77:0/2712515917 conn(0x7ff78c0d3ef0 msgr2=0x7ff78c0b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.723+0000 7ff790e4d640 1 -- 172.21.15.77:0/2712515917 shutdown_connections 2024-04-25T01:24:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.723+0000 7ff790e4d640 1 -- 172.21.15.77:0/2712515917 wait complete. 2024-04-25T01:24:19.714 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-25T01:24:19.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.830+0000 7f04ce8d6640 1 Processor -- start 2024-04-25T01:24:19.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.830+0000 7f04ce8d6640 1 -- start start 2024-04-25T01:24:19.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04ce8d6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f04c80cff20 0x7f04c80d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04ce8d6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04c80d0300 0x7f04c8145060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04ce8d6640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04c80cf5e0 0x7f04c806f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04ce8d6640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f04c80cf150 con 0x7f04c80cff20 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04ce8d6640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f04c81455a0 con 0x7f04c80cf5e0 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04ce8d6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04c8072930 con 0x7f04c80d0300 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04c7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f04c80cff20 0x7f04c80d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04c7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f04c80cff20 0x7f04c80d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49706/0 (socket says 172.21.15.77:49706) 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04c7fff640 1 -- 172.21.15.77:0/1460394061 learned_addr learned my addr 172.21.15.77:0/1460394061 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04c77fe640 1 --2- 172.21.15.77:0/1460394061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04c80d0300 0x7f04c8145060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04cce73640 1 --2- 172.21.15.77:0/1460394061 >> v2:172.21.15.121:3300/0 conn(0x7f04c80cf5e0 0x7f04c806f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04c7fff640 1 -- 172.21.15.77:0/1460394061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04c80d0300 msgr2=0x7f04c8145060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04c7fff640 1 --2- 172.21.15.77:0/1460394061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04c80d0300 0x7f04c8145060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04c7fff640 1 -- 172.21.15.77:0/1460394061 >> v2:172.21.15.121:3300/0 conn(0x7f04c80cf5e0 msgr2=0x7f04c806f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04c7fff640 1 --2- 172.21.15.77:0/1460394061 >> v2:172.21.15.121:3300/0 conn(0x7f04c80cf5e0 0x7f04c806f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.831+0000 7f04c7fff640 1 -- 172.21.15.77:0/1460394061 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c8072c10 con 0x7f04c80cff20 2024-04-25T01:24:19.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04c7fff640 1 --2- 172.21.15.77:0/1460394061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04c80cff20 0x7f04c80d1680 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f04c8068e50 tx=0x7f04b802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4ca202ef0f200b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04c6ffd640 1 -- 172.21.15.77:0/1460394061 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8037070 con 0x7f04c80cff20 2024-04-25T01:24:19.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04c6ffd640 1 -- 172.21.15.77:0/1460394061 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b8032440 con 0x7f04c80cff20 2024-04-25T01:24:19.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04c6ffd640 1 -- 172.21.15.77:0/1460394061 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8032810 con 0x7f04c80cff20 2024-04-25T01:24:19.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04ce8d6640 1 -- 172.21.15.77:0/1460394061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04c80cff20 msgr2=0x7f04c80d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04ce8d6640 1 --2- 172.21.15.77:0/1460394061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04c80cff20 0x7f04c80d1680 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f04c8068e50 tx=0x7f04b802a1e0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04ce8d6640 1 -- 172.21.15.77:0/1460394061 shutdown_connections 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04ce8d6640 1 --2- 172.21.15.77:0/1460394061 >> v2:172.21.15.121:3300/0 conn(0x7f04c80cf5e0 0x7f04c806f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04ce8d6640 1 --2- 172.21.15.77:0/1460394061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04c80d0300 0x7f04c8145060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04ce8d6640 1 --2- 172.21.15.77:0/1460394061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04c80cff20 0x7f04c80d1680 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04ce8d6640 1 -- 172.21.15.77:0/1460394061 >> 172.21.15.77:0/1460394061 conn(0x7f04c80d0b60 msgr2=0x7f04c80d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04ce8d6640 1 -- 172.21.15.77:0/1460394061 shutdown_connections 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.832+0000 7f04ce8d6640 1 -- 172.21.15.77:0/1460394061 wait complete. 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.833+0000 7f04ce8d6640 1 Processor -- start 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.834+0000 7f04ce8d6640 1 -- start start 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.834+0000 7f04ce8d6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04c80cf5e0 0x7f04c8155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.834+0000 7f04ce8d6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04c80cff20 0x7f04c8156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.834+0000 7f04ce8d6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04c80d0300 0x7f04c80caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.834+0000 7f04ce8d6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f04c80d0780 con 0x7f04c80d0300 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.834+0000 7f04ce8d6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04c80cede0 con 0x7f04c80cff20 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.834+0000 7f04ce8d6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04c80cfa40 con 0x7f04c80cf5e0 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.834+0000 7f04cce73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04c80d0300 0x7f04c80caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.834+0000 7f04cce73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04c80d0300 0x7f04c80caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49720/0 (socket says 172.21.15.77:49720) 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.834+0000 7f04cce73640 1 -- 172.21.15.77:0/891513886 learned_addr learned my addr 172.21.15.77:0/891513886 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.834+0000 7f04c77fe640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04c80cff20 0x7f04c8156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04c7fff640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04c80cf5e0 0x7f04c8155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04cce73640 1 -- 172.21.15.77:0/891513886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04c80cf5e0 msgr2=0x7f04c8155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04cce73640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04c80cf5e0 0x7f04c8155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04cce73640 1 -- 172.21.15.77:0/891513886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04c80cff20 msgr2=0x7f04c8156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04cce73640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04c80cff20 0x7f04c8156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04cce73640 1 -- 172.21.15.77:0/891513886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c8156580 con 0x7f04c80d0300 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04cce73640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04c80d0300 0x7f04c80caed0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f04c0007e90 tx=0x7f04c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04c4ff9640 1 -- 172.21.15.77:0/891513886 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c0014420 con 0x7f04c80d0300 2024-04-25T01:24:19.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04c4ff9640 1 -- 172.21.15.77:0/891513886 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04c0019070 con 0x7f04c80d0300 2024-04-25T01:24:19.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04ce8d6640 1 -- 172.21.15.77:0/891513886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c80cc240 con 0x7f04c80d0300 2024-04-25T01:24:19.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04c4ff9640 1 -- 172.21.15.77:0/891513886 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c0014820 con 0x7f04c80d0300 2024-04-25T01:24:19.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.835+0000 7f04ce8d6640 1 -- 172.21.15.77:0/891513886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c80cc800 con 0x7f04c80d0300 2024-04-25T01:24:19.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.836+0000 7f04ce8d6640 1 -- 172.21.15.77:0/891513886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c8068fb0 con 0x7f04c80d0300 2024-04-25T01:24:19.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.836+0000 7f04c4ff9640 1 -- 172.21.15.77:0/891513886 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f04c00149c0 con 0x7f04c80d0300 2024-04-25T01:24:19.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.837+0000 7f04c4ff9640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f04b406be30 0x7f04b406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.837+0000 7f04c4ff9640 1 -- 172.21.15.77:0/891513886 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f04c008b3a0 con 0x7f04c80d0300 2024-04-25T01:24:19.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.837+0000 7f04c7fff640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f04b406be30 0x7f04b406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.837+0000 7f04c7fff640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f04b406be30 0x7f04b406e2f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f04b8034000 tx=0x7f04b802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:19.841+0000 7f04c4ff9640 1 -- 172.21.15.77:0/891513886 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f04c0059850 con 0x7f04c80d0300 2024-04-25T01:24:19.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.014+0000 7f04ce8d6640 1 -- 172.21.15.77:0/891513886 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f04c806e440 con 0x7f04b406be30 2024-04-25T01:24:19.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.014+0000 7f04c4ff9640 1 -- 172.21.15.77:0/891513886 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f04c806e440 con 0x7f04b406be30 2024-04-25T01:24:19.995 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T01:24:19.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 -- 172.21.15.77:0/891513886 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f04b406be30 msgr2=0x7f04b406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f04b406be30 0x7f04b406e2f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f04b8034000 tx=0x7f04b802e000 comp rx=0 tx=0).stop 2024-04-25T01:24:19.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 -- 172.21.15.77:0/891513886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04c80d0300 msgr2=0x7f04c80caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04c80d0300 0x7f04c80caed0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f04c0007e90 tx=0x7f04c000e4b0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 -- 172.21.15.77:0/891513886 shutdown_connections 2024-04-25T01:24:19.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f04b406be30 0x7f04b406e2f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04c80d0300 0x7f04c80caed0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04c80cff20 0x7f04c8156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 --2- 172.21.15.77:0/891513886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04c80cf5e0 0x7f04c8155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 -- 172.21.15.77:0/891513886 >> 172.21.15.77:0/891513886 conn(0x7f04c80d0b60 msgr2=0x7f04c80cd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:19.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 -- 172.21.15.77:0/891513886 shutdown_connections 2024-04-25T01:24:19.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.016+0000 7f04ce8d6640 1 -- 172.21.15.77:0/891513886 wait complete. 2024-04-25T01:24:20.007 INFO:tasks.ceph:Scrubbing osd.2 2024-04-25T01:24:20.008 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:24:20.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.127+0000 7f9d6b777640 1 Processor -- start 2024-04-25T01:24:20.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.127+0000 7f9d6b777640 1 -- start start 2024-04-25T01:24:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.128+0000 7f9d6b777640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d6415e4c0 0x7f9d640d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.128+0000 7f9d6b777640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9d640d4a10 0x7f9d640b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.128+0000 7f9d6b777640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d640d2d40 0x7f9d640b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.128+0000 7f9d6b777640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f9d640d1e90 con 0x7f9d640d2d40 2024-04-25T01:24:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.128+0000 7f9d6b777640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9d640b54a0 con 0x7f9d640d4a10 2024-04-25T01:24:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.128+0000 7f9d6b777640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d640d2130 con 0x7f9d6415e4c0 2024-04-25T01:24:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.128+0000 7f9d69d14640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d640d2d40 0x7f9d640b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.128+0000 7f9d69d14640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d640d2d40 0x7f9d640b4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49728/0 (socket says 172.21.15.77:49728) 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.128+0000 7f9d69d14640 1 -- 172.21.15.77:0/3808261757 learned_addr learned my addr 172.21.15.77:0/3808261757 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.129+0000 7f9d68d12640 1 --2- 172.21.15.77:0/3808261757 >> v2:172.21.15.121:3300/0 conn(0x7f9d640d4a10 0x7f9d640b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.129+0000 7f9d69513640 1 --2- 172.21.15.77:0/3808261757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d6415e4c0 0x7f9d640d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.129+0000 7f9d69d14640 1 -- 172.21.15.77:0/3808261757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d6415e4c0 msgr2=0x7f9d640d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.129+0000 7f9d69d14640 1 --2- 172.21.15.77:0/3808261757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d6415e4c0 0x7f9d640d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.129+0000 7f9d69d14640 1 -- 172.21.15.77:0/3808261757 >> v2:172.21.15.121:3300/0 conn(0x7f9d640d4a10 msgr2=0x7f9d640b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.129+0000 7f9d69d14640 1 --2- 172.21.15.77:0/3808261757 >> v2:172.21.15.121:3300/0 conn(0x7f9d640d4a10 0x7f9d640b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.129+0000 7f9d69d14640 1 -- 172.21.15.77:0/3808261757 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d640b5780 con 0x7f9d640d2d40 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.129+0000 7f9d69d14640 1 --2- 172.21.15.77:0/3808261757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d640d2d40 0x7f9d640b4420 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9d6408c840 tx=0x7f9d5c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2405ce1be2264eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d5bfff640 1 -- 172.21.15.77:0/3808261757 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c037070 con 0x7f9d640d2d40 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d5bfff640 1 -- 172.21.15.77:0/3808261757 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d5c032440 con 0x7f9d640d2d40 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d5bfff640 1 -- 172.21.15.77:0/3808261757 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c032810 con 0x7f9d640d2d40 2024-04-25T01:24:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d6b777640 1 -- 172.21.15.77:0/3808261757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d640d2d40 msgr2=0x7f9d640b4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d6b777640 1 --2- 172.21.15.77:0/3808261757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d640d2d40 0x7f9d640b4420 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9d6408c840 tx=0x7f9d5c02a140 comp rx=0 tx=0).stop 2024-04-25T01:24:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d6b777640 1 -- 172.21.15.77:0/3808261757 shutdown_connections 2024-04-25T01:24:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d6b777640 1 --2- 172.21.15.77:0/3808261757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d640d2d40 0x7f9d640b4420 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d6b777640 1 --2- 172.21.15.77:0/3808261757 >> v2:172.21.15.121:3300/0 conn(0x7f9d640d4a10 0x7f9d640b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d6b777640 1 --2- 172.21.15.77:0/3808261757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d6415e4c0 0x7f9d640d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d6b777640 1 -- 172.21.15.77:0/3808261757 >> 172.21.15.77:0/3808261757 conn(0x7f9d640d3ef0 msgr2=0x7f9d640d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d6b777640 1 -- 172.21.15.77:0/3808261757 shutdown_connections 2024-04-25T01:24:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.130+0000 7f9d6b777640 1 -- 172.21.15.77:0/3808261757 wait complete. 2024-04-25T01:24:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.131+0000 7f9d6b777640 1 Processor -- start 2024-04-25T01:24:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.131+0000 7f9d6b777640 1 -- start start 2024-04-25T01:24:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d6b777640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d640d2d40 0x7f9d641f6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d6b777640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d640d4a10 0x7f9d64151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d6b777640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d6415e4c0 0x7f9d6414e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d6b777640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6415ea20 con 0x7f9d6415e4c0 2024-04-25T01:24:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d6b777640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6415e8a0 con 0x7f9d640d2d40 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d6b777640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d640d0900 con 0x7f9d640d4a10 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d69d14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d6415e4c0 0x7f9d6414e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d69d14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d6415e4c0 0x7f9d6414e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49744/0 (socket says 172.21.15.77:49744) 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d69d14640 1 -- 172.21.15.77:0/2773757138 learned_addr learned my addr 172.21.15.77:0/2773757138 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d69513640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d640d2d40 0x7f9d641f6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d68d12640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d640d4a10 0x7f9d64151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d69d14640 1 -- 172.21.15.77:0/2773757138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d640d4a10 msgr2=0x7f9d64151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.132+0000 7f9d69d14640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d640d4a10 0x7f9d64151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.133+0000 7f9d69d14640 1 -- 172.21.15.77:0/2773757138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d640d2d40 msgr2=0x7f9d641f6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.133+0000 7f9d69d14640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d640d2d40 0x7f9d641f6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.133+0000 7f9d69d14640 1 -- 172.21.15.77:0/2773757138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d64151570 con 0x7f9d6415e4c0 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.133+0000 7f9d68d12640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d640d4a10 0x7f9d64151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.133+0000 7f9d69d14640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d6415e4c0 0x7f9d6414e590 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0090b0 tx=0x7f9d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.133+0000 7f9d59ffb640 1 -- 172.21.15.77:0/2773757138 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c041070 con 0x7f9d6415e4c0 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.133+0000 7f9d59ffb640 1 -- 172.21.15.77:0/2773757138 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d5c032e50 con 0x7f9d6415e4c0 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.133+0000 7f9d59ffb640 1 -- 172.21.15.77:0/2773757138 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c037040 con 0x7f9d6415e4c0 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.133+0000 7f9d6b777640 1 -- 172.21.15.77:0/2773757138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d640cb210 con 0x7f9d6415e4c0 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.133+0000 7f9d6b777640 1 -- 172.21.15.77:0/2773757138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d640ccf30 con 0x7f9d6415e4c0 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.134+0000 7f9d6b777640 1 -- 172.21.15.77:0/2773757138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9d6414e2e0 con 0x7f9d6415e4c0 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.135+0000 7f9d59ffb640 1 -- 172.21.15.77:0/2773757138 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9d5c044070 con 0x7f9d6415e4c0 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.135+0000 7f9d59ffb640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9d4806be30 0x7f9d4806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.135+0000 7f9d59ffb640 1 -- 172.21.15.77:0/2773757138 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f9d5c0ab5c0 con 0x7f9d6415e4c0 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.135+0000 7f9d69513640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9d4806be30 0x7f9d4806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.135+0000 7f9d59ffb640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] conn(0x7f9d48072410 0x7f9d48074850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.135+0000 7f9d59ffb640 1 -- 172.21.15.77:0/2773757138 --> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d5c02be50 con 0x7f9d48072410 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.135+0000 7f9d59ffb640 1 -- 172.21.15.77:0/2773757138 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f9d5c0b2050 con 0x7f9d6415e4c0 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.135+0000 7f9d68d12640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] conn(0x7f9d48072410 0x7f9d48074850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.135+0000 7f9d69513640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9d4806be30 0x7f9d4806e2f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9d600073f0 tx=0x7f9d6000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.136+0000 7f9d68d12640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] conn(0x7f9d48072410 0x7f9d48074850 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.136+0000 7f9d59ffb640 1 -- 172.21.15.77:0/2773757138 <== osd.2 v2:172.21.15.77:6800/1130238186 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d5c02be50 con 0x7f9d48072410 2024-04-25T01:24:20.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.149+0000 7f9d6b777640 1 -- 172.21.15.77:0/2773757138 --> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9d640cafc0 con 0x7f9d48072410 2024-04-25T01:24:20.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.149+0000 7f9d59ffb640 1 -- 172.21.15.77:0/2773757138 <== osd.2 v2:172.21.15.77:6800/1130238186 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9d640cafc0 con 0x7f9d48072410 2024-04-25T01:24:20.130 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:24:20.130 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:24:20.130 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:24:20.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.150+0000 7f9d6b777640 1 -- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] conn(0x7f9d48072410 msgr2=0x7f9d48074850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.150+0000 7f9d6b777640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:6800/1130238186,v1:172.21.15.77:6801/1130238186] conn(0x7f9d48072410 0x7f9d48074850 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.150+0000 7f9d6b777640 1 -- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9d4806be30 msgr2=0x7f9d4806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.150+0000 7f9d6b777640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f9d4806be30 0x7f9d4806e2f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9d600073f0 tx=0x7f9d6000b040 comp rx=0 tx=0).stop 2024-04-25T01:24:20.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.150+0000 7f9d6b777640 1 -- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d6415e4c0 msgr2=0x7f9d6414e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.150+0000 7f9d6b777640 1 --2- 172.21.15.77:0/2773757138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d6415e4c0 0x7f9d6414e590 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0090b0 tx=0x7f9d5c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.150+0000 7f9d69d14640 1 -- 172.21.15.77:0/2773757138 reap_dead start 2024-04-25T01:24:20.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.150+0000 7f9d6b777640 1 -- 172.21.15.77:0/2773757138 shutdown_connections 2024-04-25T01:24:20.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.150+0000 7f9d6b777640 1 -- 172.21.15.77:0/2773757138 >> 172.21.15.77:0/2773757138 conn(0x7f9d640d3ef0 msgr2=0x7f9d640b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:20.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.150+0000 7f9d6b777640 1 -- 172.21.15.77:0/2773757138 shutdown_connections 2024-04-25T01:24:20.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.150+0000 7f9d6b777640 1 -- 172.21.15.77:0/2773757138 wait complete. 2024-04-25T01:24:20.141 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-25T01:24:20.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.258+0000 7ffa0f596640 1 Processor -- start 2024-04-25T01:24:20.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.258+0000 7ffa0f596640 1 -- start start 2024-04-25T01:24:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.259+0000 7ffa0f596640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa08143d20 0x7ffa0815ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.259+0000 7ffa0f596640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa08070e60 con 0x7ffa0806ed60 2024-04-25T01:24:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.259+0000 7ffa0f596640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa08071140 con 0x7ffa08143940 2024-04-25T01:24:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.259+0000 7ffa0f596640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7ffa08071420 con 0x7ffa08143d20 2024-04-25T01:24:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.259+0000 7ffa0cb31640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa08143d20 0x7ffa0815ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.259+0000 7ffa0db33640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ffa0806ed60 0x7ffa0815f230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38350/0 (socket says 172.21.15.77:38350) 2024-04-25T01:24:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.259+0000 7ffa0db33640 1 -- 172.21.15.77:0/2050605087 learned_addr learned my addr 172.21.15.77:0/2050605087 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.259+0000 7ffa0cb31640 1 -- 172.21.15.77:0/2050605087 >> v1:172.21.15.121:6789/0 conn(0x7ffa08143940 legacy=0x7ffa0806e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.259+0000 7ffa0cb31640 1 -- 172.21.15.77:0/2050605087 >> v1:172.21.15.77:6789/0 conn(0x7ffa0806ed60 legacy=0x7ffa0815f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.259+0000 7ffa0cb31640 1 -- 172.21.15.77:0/2050605087 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa08071700 con 0x7ffa08143d20 2024-04-25T01:24:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.259+0000 7ff9fffff640 1 -- 172.21.15.77:0/2050605087 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3909490056 0 0) 0x7ffa08070e60 con 0x7ffa0806ed60 2024-04-25T01:24:20.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.260+0000 7ffa0cb31640 1 --2- 172.21.15.77:0/2050605087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08143d20 0x7ffa0815ecf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ffa040097d0 tx=0x7ffa0402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a4b145ce521d348 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.260+0000 7ff9fffff640 1 -- 172.21.15.77:0/2050605087 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa04037070 con 0x7ffa08143d20 2024-04-25T01:24:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.260+0000 7ff9fffff640 1 -- 172.21.15.77:0/2050605087 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa04032440 con 0x7ffa08143d20 2024-04-25T01:24:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.260+0000 7ff9fffff640 1 -- 172.21.15.77:0/2050605087 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa04032810 con 0x7ffa08143d20 2024-04-25T01:24:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.260+0000 7ffa0f596640 1 -- 172.21.15.77:0/2050605087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08143d20 msgr2=0x7ffa0815ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.260+0000 7ffa0f596640 1 --2- 172.21.15.77:0/2050605087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08143d20 0x7ffa0815ecf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ffa040097d0 tx=0x7ffa0402a210 comp rx=0 tx=0).stop 2024-04-25T01:24:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.260+0000 7ffa0f596640 1 -- 172.21.15.77:0/2050605087 shutdown_connections 2024-04-25T01:24:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.260+0000 7ffa0f596640 1 --2- 172.21.15.77:0/2050605087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08143d20 0x7ffa0815ecf0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.260+0000 7ffa0f596640 1 -- 172.21.15.77:0/2050605087 >> 172.21.15.77:0/2050605087 conn(0x7ffa080b1820 msgr2=0x7ffa080b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.260+0000 7ffa0f596640 1 -- 172.21.15.77:0/2050605087 shutdown_connections 2024-04-25T01:24:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.260+0000 7ffa0f596640 1 -- 172.21.15.77:0/2050605087 wait complete. 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.262+0000 7ffa0f596640 1 Processor -- start 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.262+0000 7ffa0f596640 1 -- start start 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.262+0000 7ffa0f596640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa0806ed60 0x7ffa080ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.262+0000 7ffa0f596640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08143940 0x7ffa0815d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.262+0000 7ffa0f596640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa08143d20 0x7ffa0815de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.262+0000 7ffa0f596640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa08165d30 con 0x7ffa08143940 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.262+0000 7ffa0f596640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffa08072e80 con 0x7ffa0806ed60 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.262+0000 7ffa0f596640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffa08165eb0 con 0x7ffa08143d20 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.262+0000 7ffa0cb31640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08143940 0x7ffa0815d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.262+0000 7ffa0cb31640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08143940 0x7ffa0815d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49762/0 (socket says 172.21.15.77:49762) 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.262+0000 7ffa0cb31640 1 -- 172.21.15.77:0/4179429458 learned_addr learned my addr 172.21.15.77:0/4179429458 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ffa0d332640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa0806ed60 0x7ffa080ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ffa0cb31640 1 -- 172.21.15.77:0/4179429458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa08143d20 msgr2=0x7ffa0815de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ffa0cb31640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa08143d20 0x7ffa0815de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ffa0cb31640 1 -- 172.21.15.77:0/4179429458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa0806ed60 msgr2=0x7ffa080ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ffa0cb31640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa0806ed60 0x7ffa080ce3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ffa0cb31640 1 -- 172.21.15.77:0/4179429458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0815aa80 con 0x7ffa08143940 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ffa0d332640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa0806ed60 0x7ffa080ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ffa0cb31640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08143940 0x7ffa0815d8c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffa04035040 tx=0x7ffa040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ff9fdffb640 1 -- 172.21.15.77:0/4179429458 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa04042070 con 0x7ffa08143940 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ff9fdffb640 1 -- 172.21.15.77:0/4179429458 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa04004240 con 0x7ffa08143940 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ffa0f596640 1 -- 172.21.15.77:0/4179429458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa081599e0 con 0x7ffa08143940 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ff9fdffb640 1 -- 172.21.15.77:0/4179429458 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa04037040 con 0x7ffa08143940 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.263+0000 7ffa0f596640 1 -- 172.21.15.77:0/4179429458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa08159fa0 con 0x7ffa08143940 2024-04-25T01:24:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.264+0000 7ffa0f596640 1 -- 172.21.15.77:0/4179429458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa0806cc30 con 0x7ffa08143940 2024-04-25T01:24:20.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.265+0000 7ff9fdffb640 1 -- 172.21.15.77:0/4179429458 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ffa04030770 con 0x7ffa08143940 2024-04-25T01:24:20.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.265+0000 7ff9fdffb640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff9ec06be30 0x7ff9ec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.265+0000 7ff9fdffb640 1 -- 172.21.15.77:0/4179429458 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7ffa040ab9e0 con 0x7ffa08143940 2024-04-25T01:24:20.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.265+0000 7ffa0d332640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff9ec06be30 0x7ff9ec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.265+0000 7ffa0d332640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff9ec06be30 0x7ff9ec06e2f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8003eb0 tx=0x7ff9f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.269+0000 7ff9fdffb640 1 -- 172.21.15.77:0/4179429458 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffa04032e20 con 0x7ffa08143940 2024-04-25T01:24:20.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.441+0000 7ffa0f596640 1 -- 172.21.15.77:0/4179429458 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7ffa080cf5c0 con 0x7ff9ec06be30 2024-04-25T01:24:20.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.441+0000 7ff9fdffb640 1 -- 172.21.15.77:0/4179429458 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ffa080cf5c0 con 0x7ff9ec06be30 2024-04-25T01:24:20.422 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T01:24:20.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 -- 172.21.15.77:0/4179429458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff9ec06be30 msgr2=0x7ff9ec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff9ec06be30 0x7ff9ec06e2f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8003eb0 tx=0x7ff9f8009040 comp rx=0 tx=0).stop 2024-04-25T01:24:20.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 -- 172.21.15.77:0/4179429458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08143940 msgr2=0x7ffa0815d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08143940 0x7ffa0815d8c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffa04035040 tx=0x7ffa040046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 -- 172.21.15.77:0/4179429458 shutdown_connections 2024-04-25T01:24:20.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7ff9ec06be30 0x7ff9ec06e2f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08143940 0x7ffa0815d8c0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa08143d20 0x7ffa0815de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 --2- 172.21.15.77:0/4179429458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa0806ed60 0x7ffa080ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 -- 172.21.15.77:0/4179429458 >> 172.21.15.77:0/4179429458 conn(0x7ffa080b1820 msgr2=0x7ffa0806e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:20.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 -- 172.21.15.77:0/4179429458 shutdown_connections 2024-04-25T01:24:20.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.443+0000 7ffa0f596640 1 -- 172.21.15.77:0/4179429458 wait complete. 2024-04-25T01:24:20.434 INFO:tasks.ceph:Scrubbing osd.3 2024-04-25T01:24:20.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:24:20.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.555+0000 7f8723ad4640 1 Processor -- start 2024-04-25T01:24:20.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.555+0000 7f8723ad4640 1 -- start start 2024-04-25T01:24:20.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.556+0000 7f8723ad4640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f871c15ae90 0x7f871c0ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.556+0000 7f8723ad4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f871c0ce470 0x7f871c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.556+0000 7f8723ad4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f871c072480 con 0x7f871c06e2f0 2024-04-25T01:24:20.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.556+0000 7f8723ad4640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f871c0cf700 con 0x7f871c15ae90 2024-04-25T01:24:20.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.556+0000 7f8723ad4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f871c0cd6b0 con 0x7f871c0ce470 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.556+0000 7f8722071640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f871c06e2f0 0x7f871c15e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38356/0 (socket says 172.21.15.77:38356) 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.556+0000 7f8722071640 1 -- 172.21.15.77:0/1006668399 learned_addr learned my addr 172.21.15.77:0/1006668399 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.556+0000 7f8721870640 1 --2- 172.21.15.77:0/1006668399 >> v2:172.21.15.121:3300/0 conn(0x7f871c15ae90 0x7f871c0ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.556+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2405300749 0 0) 0x7f871c072480 con 0x7f871c06e2f0 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.556+0000 7f872106f640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f871c0ce470 0x7f871c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8708005730 con 0x7f871c06e2f0 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 837044055 0 0) 0x7f8708005730 con 0x7f871c06e2f0 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f871c0ce470 msgr2=0x7f871c0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f871c0ce470 0x7f871c0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 >> v2:172.21.15.121:3300/0 conn(0x7f871c15ae90 msgr2=0x7f871c0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 --2- 172.21.15.77:0/1006668399 >> v2:172.21.15.121:3300/0 conn(0x7f871c15ae90 0x7f871c0ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871c0cd990 con 0x7f871c06e2f0 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f8714001670 con 0x7f871c06e2f0 2024-04-25T01:24:20.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 >> v1:172.21.15.77:6789/0 conn(0x7f871c06e2f0 legacy=0x7f871c15e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8708005bc0 0x7f8708005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f871c0ce470 0x7f8708006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8708006f50 0x7f87080073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f871c0cf700 con 0x7f8708005bc0 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f871c0cd6b0 con 0x7f8708006f50 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f870800dbd0 con 0x7f871c0ce470 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f8721870640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8708005bc0 0x7f8708005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f8722071640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8708006f50 0x7f87080073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f872106f640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f871c0ce470 0x7f8708006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f8721870640 1 -- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f871c0ce470 msgr2=0x7f8708006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f8721870640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f871c0ce470 0x7f8708006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f8721870640 1 -- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8708006f50 msgr2=0x7f87080073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f8721870640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8708006f50 0x7f87080073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f8721870640 1 -- 172.21.15.77:0/1006668399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8708005730 con 0x7f8708005bc0 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.557+0000 7f8722071640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8708006f50 0x7f87080073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.558+0000 7f8721870640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8708005bc0 0x7f8708005fe0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f8718009fc0 tx=0x7f8708007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950374ef57d8c020 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.558+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718009c10 con 0x7f8708005bc0 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.558+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8718036070 con 0x7f8708005bc0 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.558+0000 7f872086e640 1 -- 172.21.15.77:0/1006668399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871802d070 con 0x7f8708005bc0 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.558+0000 7f8723ad4640 1 -- 172.21.15.77:0/1006668399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8708005bc0 msgr2=0x7f8708005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.558+0000 7f8723ad4640 1 --2- 172.21.15.77:0/1006668399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8708005bc0 0x7f8708005fe0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f8718009fc0 tx=0x7f8708007a40 comp rx=0 tx=0).stop 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.558+0000 7f8722071640 1 -- 172.21.15.77:0/1006668399 reap_dead start 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.559+0000 7f8723ad4640 1 -- 172.21.15.77:0/1006668399 shutdown_connections 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.559+0000 7f8723ad4640 1 -- 172.21.15.77:0/1006668399 >> 172.21.15.77:0/1006668399 conn(0x7f871c0cfae0 msgr2=0x7f871c0cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.559+0000 7f8723ad4640 1 -- 172.21.15.77:0/1006668399 shutdown_connections 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.559+0000 7f8723ad4640 1 -- 172.21.15.77:0/1006668399 wait complete. 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.560+0000 7f8723ad4640 1 Processor -- start 2024-04-25T01:24:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.560+0000 7f8723ad4640 1 -- start start 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.560+0000 7f8723ad4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8708005bc0 0x7f871c13e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.560+0000 7f8723ad4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f871c153db0 0x7f871c14a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.560+0000 7f8723ad4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f871c15a3e0 0x7f871c14abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.560+0000 7f8723ad4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f871c0cf0d0 con 0x7f8708005bc0 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.560+0000 7f8723ad4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f871c0cdf50 con 0x7f871c153db0 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.560+0000 7f8723ad4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f871c0cf250 con 0x7f871c15a3e0 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.560+0000 7f8721870640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8708005bc0 0x7f871c13e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f8721870640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8708005bc0 0x7f871c13e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49768/0 (socket says 172.21.15.77:49768) 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f8721870640 1 -- 172.21.15.77:0/840694228 learned_addr learned my addr 172.21.15.77:0/840694228 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f872106f640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f871c153db0 0x7f871c14a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f8721870640 1 -- 172.21.15.77:0/840694228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f871c15a3e0 msgr2=0x7f871c14abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f8721870640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f871c15a3e0 0x7f871c14abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f8721870640 1 -- 172.21.15.77:0/840694228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f871c153db0 msgr2=0x7f871c14a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f8721870640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f871c153db0 0x7f871c14a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f8721870640 1 -- 172.21.15.77:0/840694228 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871c159a70 con 0x7f8708005bc0 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f8721870640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8708005bc0 0x7f871c13e780 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f871800e230 tx=0x7f8718009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f87127fc640 1 -- 172.21.15.77:0/840694228 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718009c10 con 0x7f8708005bc0 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f87127fc640 1 -- 172.21.15.77:0/840694228 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8718003b90 con 0x7f8708005bc0 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f8723ad4640 1 -- 172.21.15.77:0/840694228 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f871c159d50 con 0x7f8708005bc0 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f87127fc640 1 -- 172.21.15.77:0/840694228 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718004020 con 0x7f8708005bc0 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.561+0000 7f8723ad4640 1 -- 172.21.15.77:0/840694228 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f871c153070 con 0x7f8708005bc0 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.562+0000 7f8723ad4640 1 -- 172.21.15.77:0/840694228 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f871c068fb0 con 0x7f8708005bc0 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.562+0000 7f87127fc640 1 -- 172.21.15.77:0/840694228 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8718005d10 con 0x7f8708005bc0 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.563+0000 7f87127fc640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f870006be30 0x7f870006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.563+0000 7f87127fc640 1 -- 172.21.15.77:0/840694228 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f87180ac240 con 0x7f8708005bc0 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.563+0000 7f872106f640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f870006be30 0x7f870006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.563+0000 7f87127fc640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] conn(0x7f8700072410 0x7f8700074850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.563+0000 7f87127fc640 1 -- 172.21.15.77:0/840694228 --> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f871800df90 con 0x7f8700072410 2024-04-25T01:24:20.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.563+0000 7f87127fc640 1 -- 172.21.15.77:0/840694228 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f87180b3050 con 0x7f8708005bc0 2024-04-25T01:24:20.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.563+0000 7f8722071640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] conn(0x7f8700072410 0x7f8700074850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.563+0000 7f872106f640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f870006be30 0x7f870006e2f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f871c15efe0 tx=0x7f870c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.563+0000 7f8722071640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] conn(0x7f8700072410 0x7f8700074850 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.564+0000 7f87127fc640 1 -- 172.21.15.77:0/840694228 <== osd.3 v2:172.21.15.77:6806/3656221607 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f871800df90 con 0x7f8700072410 2024-04-25T01:24:20.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.577+0000 7f8723ad4640 1 -- 172.21.15.77:0/840694228 --> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f871c06ca90 con 0x7f8700072410 2024-04-25T01:24:20.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.577+0000 7f87127fc640 1 -- 172.21.15.77:0/840694228 <== osd.3 v2:172.21.15.77:6806/3656221607 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f871c06ca90 con 0x7f8700072410 2024-04-25T01:24:20.558 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:24:20.558 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:24:20.558 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:24:20.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.577+0000 7f8723ad4640 1 -- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] conn(0x7f8700072410 msgr2=0x7f8700074850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.577+0000 7f8723ad4640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:6806/3656221607,v1:172.21.15.77:6809/3656221607] conn(0x7f8700072410 0x7f8700074850 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.578+0000 7f8723ad4640 1 -- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f870006be30 msgr2=0x7f870006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.578+0000 7f8723ad4640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f870006be30 0x7f870006e2f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f871c15efe0 tx=0x7f870c008040 comp rx=0 tx=0).stop 2024-04-25T01:24:20.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.578+0000 7f8723ad4640 1 -- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8708005bc0 msgr2=0x7f871c13e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.578+0000 7f8723ad4640 1 --2- 172.21.15.77:0/840694228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8708005bc0 0x7f871c13e780 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f871800e230 tx=0x7f8718009750 comp rx=0 tx=0).stop 2024-04-25T01:24:20.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.578+0000 7f8722071640 1 -- 172.21.15.77:0/840694228 reap_dead start 2024-04-25T01:24:20.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.578+0000 7f8723ad4640 1 -- 172.21.15.77:0/840694228 shutdown_connections 2024-04-25T01:24:20.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.578+0000 7f8723ad4640 1 -- 172.21.15.77:0/840694228 >> 172.21.15.77:0/840694228 conn(0x7f871c0cfae0 msgr2=0x7f871c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:20.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.578+0000 7f8723ad4640 1 -- 172.21.15.77:0/840694228 shutdown_connections 2024-04-25T01:24:20.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.578+0000 7f8723ad4640 1 -- 172.21.15.77:0/840694228 wait complete. 2024-04-25T01:24:20.569 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-25T01:24:20.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.686+0000 7f4088fc1640 1 Processor -- start 2024-04-25T01:24:20.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.686+0000 7f4088fc1640 1 -- start start 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.687+0000 7f4088fc1640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4084154ac0 0x7f40840711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.687+0000 7f4088fc1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4084071730 0x7f408406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.687+0000 7f4088fc1640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408406d790 con 0x7f408415b5f0 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.687+0000 7f4088fc1640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f408415b9d0 con 0x7f4084154ac0 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.687+0000 7f4088fc1640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f408406da70 con 0x7f4084071730 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.687+0000 7f408259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f408415b5f0 0x7f40840cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38360/0 (socket says 172.21.15.77:38360) 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.687+0000 7f408259c640 1 -- 172.21.15.77:0/2779342038 learned_addr learned my addr 172.21.15.77:0/2779342038 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.687+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 128021139 0 0) 0x7f408406d790 con 0x7f408415b5f0 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.687+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f406c003520 con 0x7f408415b5f0 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.687+0000 7f4081d9b640 1 --2- 172.21.15.77:0/2779342038 >> v2:172.21.15.121:3300/0 conn(0x7f4084154ac0 0x7f40840711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.687+0000 7f4082d9d640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4084071730 0x7f408406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3088557115 0 0) 0x7f406c003520 con 0x7f408415b5f0 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4084071730 msgr2=0x7f408406ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4084071730 0x7f408406ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 >> v2:172.21.15.121:3300/0 conn(0x7f4084154ac0 msgr2=0x7f40840711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 --2- 172.21.15.77:0/2779342038 >> v2:172.21.15.121:3300/0 conn(0x7f4084154ac0 0x7f40840711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408406dd50 con 0x7f408415b5f0 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f40740019f0 con 0x7f408415b5f0 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f4082d9d640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4084071730 0x7f408406ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 >> v1:172.21.15.77:6789/0 conn(0x7f408415b5f0 legacy=0x7f40840cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f406c0039b0 0x7f406c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 0x7f406c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4084071730 0x7f406c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f408415b9d0 con 0x7f406c004800 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f408406da70 con 0x7f406c0039b0 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f406c00b9c0 con 0x7f4084071730 2024-04-25T01:24:20.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f4081d9b640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 0x7f406c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f408259c640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f406c0039b0 0x7f406c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f4082d9d640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4084071730 0x7f406c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f4081d9b640 1 -- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4084071730 msgr2=0x7f406c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f4081d9b640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4084071730 0x7f406c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f4081d9b640 1 -- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f406c0039b0 msgr2=0x7f406c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f4081d9b640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f406c0039b0 0x7f406c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.688+0000 7f4081d9b640 1 -- 172.21.15.77:0/2779342038 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c003520 con 0x7f406c004800 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.689+0000 7f4081d9b640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 0x7f406c004c80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4078009af0 tx=0x7f406c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab140ef4b7a727c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.689+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078036070 con 0x7f406c004800 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.689+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4078031400 con 0x7f406c004800 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.689+0000 7f408159a640 1 -- 172.21.15.77:0/2779342038 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078009690 con 0x7f406c004800 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.689+0000 7f4088fc1640 1 -- 172.21.15.77:0/2779342038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 msgr2=0x7f406c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.689+0000 7f4088fc1640 1 --2- 172.21.15.77:0/2779342038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 0x7f406c004c80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4078009af0 tx=0x7f406c005830 comp rx=0 tx=0).stop 2024-04-25T01:24:20.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.689+0000 7f4082d9d640 1 -- 172.21.15.77:0/2779342038 reap_dead start 2024-04-25T01:24:20.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.690+0000 7f4088fc1640 1 -- 172.21.15.77:0/2779342038 shutdown_connections 2024-04-25T01:24:20.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.690+0000 7f4088fc1640 1 -- 172.21.15.77:0/2779342038 >> 172.21.15.77:0/2779342038 conn(0x7f40840b1820 msgr2=0x7f40840b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:20.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.690+0000 7f4088fc1640 1 -- 172.21.15.77:0/2779342038 shutdown_connections 2024-04-25T01:24:20.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.690+0000 7f4088fc1640 1 -- 172.21.15.77:0/2779342038 wait complete. 2024-04-25T01:24:20.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.691+0000 7f4088fc1640 1 Processor -- start 2024-04-25T01:24:20.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.691+0000 7f4088fc1640 1 -- start start 2024-04-25T01:24:20.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f4088fc1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 0x7f40840cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f4088fc1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f408415f300 0x7f40841573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f4088fc1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f408406e430 0x7f408406e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f4088fc1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f408406ce30 con 0x7f406c004800 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f4088fc1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f408406ccb0 con 0x7f408415f300 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f4088fc1640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f408406cfb0 con 0x7f408406e430 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f408259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 0x7f40840cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f408259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 0x7f40840cb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49780/0 (socket says 172.21.15.77:49780) 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f408259c640 1 -- 172.21.15.77:0/1196442729 learned_addr learned my addr 172.21.15.77:0/1196442729 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f408259c640 1 -- 172.21.15.77:0/1196442729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f408406e430 msgr2=0x7f408406e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f408259c640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f408406e430 0x7f408406e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f408259c640 1 -- 172.21.15.77:0/1196442729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f408415f300 msgr2=0x7f40841573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f408259c640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f408415f300 0x7f40841573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f408259c640 1 -- 172.21.15.77:0/1196442729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40841f1fb0 con 0x7f406c004800 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f408259c640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 0x7f40840cb2a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f4074007970 tx=0x7f407400a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f4072ffd640 1 -- 172.21.15.77:0/1196442729 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074017020 con 0x7f406c004800 2024-04-25T01:24:20.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f4072ffd640 1 -- 172.21.15.77:0/1196442729 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f407400ae00 con 0x7f406c004800 2024-04-25T01:24:20.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f4088fc1640 1 -- 172.21.15.77:0/1196442729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408415f7e0 con 0x7f406c004800 2024-04-25T01:24:20.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f4072ffd640 1 -- 172.21.15.77:0/1196442729 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074013610 con 0x7f406c004800 2024-04-25T01:24:20.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.692+0000 7f4088fc1640 1 -- 172.21.15.77:0/1196442729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f408415fda0 con 0x7f406c004800 2024-04-25T01:24:20.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.693+0000 7f4088fc1640 1 -- 172.21.15.77:0/1196442729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4084068fb0 con 0x7f406c004800 2024-04-25T01:24:20.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.694+0000 7f4072ffd640 1 -- 172.21.15.77:0/1196442729 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f407401a020 con 0x7f406c004800 2024-04-25T01:24:20.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.694+0000 7f4072ffd640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f406406be60 0x7f406406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.694+0000 7f4081d9b640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f406406be60 0x7f406406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.694+0000 7f4072ffd640 1 -- 172.21.15.77:0/1196442729 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f407408b840 con 0x7f406c004800 2024-04-25T01:24:20.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.695+0000 7f4081d9b640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f406406be60 0x7f406406e320 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f4078033000 tx=0x7f407802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.699+0000 7f4072ffd640 1 -- 172.21.15.77:0/1196442729 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4074021020 con 0x7f406c004800 2024-04-25T01:24:20.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.871+0000 7f4088fc1640 1 -- 172.21.15.77:0/1196442729 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f40840b2980 con 0x7f406406be60 2024-04-25T01:24:20.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.871+0000 7f4072ffd640 1 -- 172.21.15.77:0/1196442729 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f40840b2980 con 0x7f406406be60 2024-04-25T01:24:20.852 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 3 to deep-scrub 2024-04-25T01:24:20.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.873+0000 7f4088fc1640 1 -- 172.21.15.77:0/1196442729 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f406406be60 msgr2=0x7f406406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.873+0000 7f4088fc1640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f406406be60 0x7f406406e320 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f4078033000 tx=0x7f407802e000 comp rx=0 tx=0).stop 2024-04-25T01:24:20.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.873+0000 7f4088fc1640 1 -- 172.21.15.77:0/1196442729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 msgr2=0x7f40840cb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.873+0000 7f4088fc1640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 0x7f40840cb2a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f4074007970 tx=0x7f407400a750 comp rx=0 tx=0).stop 2024-04-25T01:24:20.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.873+0000 7f4088fc1640 1 -- 172.21.15.77:0/1196442729 shutdown_connections 2024-04-25T01:24:20.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.873+0000 7f4088fc1640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f406406be60 0x7f406406e320 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.873+0000 7f4088fc1640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f408406e430 0x7f408406e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.873+0000 7f4088fc1640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f408415f300 0x7f40841573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.873+0000 7f4088fc1640 1 --2- 172.21.15.77:0/1196442729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f406c004800 0x7f40840cb2a0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.873+0000 7f4088fc1640 1 -- 172.21.15.77:0/1196442729 >> 172.21.15.77:0/1196442729 conn(0x7f40840b1820 msgr2=0x7f408415a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:20.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.874+0000 7f4088fc1640 1 -- 172.21.15.77:0/1196442729 shutdown_connections 2024-04-25T01:24:20.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.874+0000 7f4088fc1640 1 -- 172.21.15.77:0/1196442729 wait complete. 2024-04-25T01:24:20.865 INFO:tasks.ceph:Scrubbing osd.4 2024-04-25T01:24:20.865 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:24:20.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.986+0000 7f457e334640 1 Processor -- start 2024-04-25T01:24:20.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.986+0000 7f457e334640 1 -- start start 2024-04-25T01:24:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.986+0000 7f457e334640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f457815ae90 0x7f45780ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.986+0000 7f457e334640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f45780ce470 0x7f45780cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.986+0000 7f457e334640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f457806e2f0 0x7f457806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.986+0000 7f457e334640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f45780cf700 con 0x7f457815ae90 2024-04-25T01:24:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.986+0000 7f457e334640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f45780cd6b0 con 0x7f45780ce470 2024-04-25T01:24:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.986+0000 7f457e334640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f45780cd990 con 0x7f457806e2f0 2024-04-25T01:24:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.987+0000 7f4577fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f457815ae90 0x7f45780ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.987+0000 7f4577fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f457815ae90 0x7f45780ceb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49796/0 (socket says 172.21.15.77:49796) 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.987+0000 7f4577fff640 1 -- 172.21.15.77:0/3639708865 learned_addr learned my addr 172.21.15.77:0/3639708865 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.987+0000 7f45777fe640 1 --2- 172.21.15.77:0/3639708865 >> v2:172.21.15.121:3300/0 conn(0x7f45780ce470 0x7f45780cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.987+0000 7f457c8d1640 1 --2- 172.21.15.77:0/3639708865 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f457806e2f0 0x7f457806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.987+0000 7f4577fff640 1 -- 172.21.15.77:0/3639708865 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f457806e2f0 msgr2=0x7f457806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.987+0000 7f4577fff640 1 --2- 172.21.15.77:0/3639708865 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f457806e2f0 0x7f457806e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.987+0000 7f4577fff640 1 -- 172.21.15.77:0/3639708865 >> v2:172.21.15.121:3300/0 conn(0x7f45780ce470 msgr2=0x7f45780cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.987+0000 7f4577fff640 1 --2- 172.21.15.77:0/3639708865 >> v2:172.21.15.121:3300/0 conn(0x7f45780ce470 0x7f45780cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.987+0000 7f4577fff640 1 -- 172.21.15.77:0/3639708865 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45780cdc70 con 0x7f457815ae90 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.987+0000 7f457c8d1640 1 --2- 172.21.15.77:0/3639708865 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f457806e2f0 0x7f457806e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f4577fff640 1 --2- 172.21.15.77:0/3639708865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f457815ae90 0x7f45780ceb90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f457806c410 tx=0x7f456002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa843738f27cdd93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f4576ffd640 1 -- 172.21.15.77:0/3639708865 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560037070 con 0x7f457815ae90 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f4576ffd640 1 -- 172.21.15.77:0/3639708865 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4560032440 con 0x7f457815ae90 2024-04-25T01:24:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f4576ffd640 1 -- 172.21.15.77:0/3639708865 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560032810 con 0x7f457815ae90 2024-04-25T01:24:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f457e334640 1 -- 172.21.15.77:0/3639708865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f457815ae90 msgr2=0x7f45780ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f457e334640 1 --2- 172.21.15.77:0/3639708865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f457815ae90 0x7f45780ceb90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f457806c410 tx=0x7f456002a1e0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f457e334640 1 -- 172.21.15.77:0/3639708865 shutdown_connections 2024-04-25T01:24:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f457e334640 1 --2- 172.21.15.77:0/3639708865 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f457806e2f0 0x7f457806e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f457e334640 1 --2- 172.21.15.77:0/3639708865 >> v2:172.21.15.121:3300/0 conn(0x7f45780ce470 0x7f45780cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f457e334640 1 --2- 172.21.15.77:0/3639708865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f457815ae90 0x7f45780ceb90 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f457e334640 1 -- 172.21.15.77:0/3639708865 >> 172.21.15.77:0/3639708865 conn(0x7f45780cfae0 msgr2=0x7f45780cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f457e334640 1 -- 172.21.15.77:0/3639708865 shutdown_connections 2024-04-25T01:24:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.988+0000 7f457e334640 1 -- 172.21.15.77:0/3639708865 wait complete. 2024-04-25T01:24:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.989+0000 7f457e334640 1 Processor -- start 2024-04-25T01:24:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.990+0000 7f457e334640 1 -- start start 2024-04-25T01:24:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.990+0000 7f457e334640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f457806e2f0 0x7f457814e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.990+0000 7f457e334640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45780ce470 0x7f457814c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.990+0000 7f457e334640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f457815ae90 0x7f457814c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.990+0000 7f457e334640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f457815e800 con 0x7f45780ce470 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.990+0000 7f457e334640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f457806cf40 con 0x7f457815ae90 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.990+0000 7f457e334640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f457815e980 con 0x7f457806e2f0 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.990+0000 7f45777fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45780ce470 0x7f457814c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.990+0000 7f45777fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45780ce470 0x7f457814c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49810/0 (socket says 172.21.15.77:49810) 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.990+0000 7f45777fe640 1 -- 172.21.15.77:0/665786429 learned_addr learned my addr 172.21.15.77:0/665786429 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f457c8d1640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f457815ae90 0x7f457814c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f4577fff640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f457806e2f0 0x7f457814e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f45777fe640 1 -- 172.21.15.77:0/665786429 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f457806e2f0 msgr2=0x7f457814e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f45777fe640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f457806e2f0 0x7f457814e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f45777fe640 1 -- 172.21.15.77:0/665786429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f457815ae90 msgr2=0x7f457814c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f45777fe640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f457815ae90 0x7f457814c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f45777fe640 1 -- 172.21.15.77:0/665786429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f457814e6b0 con 0x7f45780ce470 2024-04-25T01:24:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f45777fe640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45780ce470 0x7f457814c000 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f456800d800 tx=0x7f456800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f4574ff9640 1 -- 172.21.15.77:0/665786429 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456800b6c0 con 0x7f45780ce470 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f4574ff9640 1 -- 172.21.15.77:0/665786429 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4568019070 con 0x7f45780ce470 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f4574ff9640 1 -- 172.21.15.77:0/665786429 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456800ba80 con 0x7f45780ce470 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f457e334640 1 -- 172.21.15.77:0/665786429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f457814e990 con 0x7f45780ce470 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.991+0000 7f457e334640 1 -- 172.21.15.77:0/665786429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f457814ef50 con 0x7f45780ce470 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.992+0000 7f457e334640 1 -- 172.21.15.77:0/665786429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f457813be00 con 0x7f45780ce470 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.992+0000 7f4574ff9640 1 -- 172.21.15.77:0/665786429 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f456800bc20 con 0x7f45780ce470 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.993+0000 7f4574ff9640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f455806be30 0x7f455806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.993+0000 7f4577fff640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f455806be30 0x7f455806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.993+0000 7f4574ff9640 1 -- 172.21.15.77:0/665786429 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f456808b1a0 con 0x7f45780ce470 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.993+0000 7f4574ff9640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] conn(0x7f4558072410 0x7f4558074850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.993+0000 7f4574ff9640 1 -- 172.21.15.77:0/665786429 --> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4558074f20 con 0x7f4558072410 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.993+0000 7f4574ff9640 1 -- 172.21.15.77:0/665786429 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f4568092050 con 0x7f45780ce470 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.993+0000 7f457c8d1640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] conn(0x7f4558072410 0x7f4558074850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.993+0000 7f4577fff640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f455806be30 0x7f455806e2f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4560034000 tx=0x7f456002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.994+0000 7f457c8d1640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] conn(0x7f4558072410 0x7f4558074850 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:20.994+0000 7f4574ff9640 1 -- 172.21.15.77:0/665786429 <== osd.4 v2:172.21.15.121:6803/3650663429 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4558074f20 con 0x7f4558072410 2024-04-25T01:24:20.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.007+0000 7f457e334640 1 -- 172.21.15.77:0/665786429 --> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4578068fb0 con 0x7f4558072410 2024-04-25T01:24:20.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f4574ff9640 1 -- 172.21.15.77:0/665786429 <== osd.4 v2:172.21.15.121:6803/3650663429 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4578068fb0 con 0x7f4558072410 2024-04-25T01:24:20.989 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:24:20.989 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:24:20.989 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:24:20.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f457e334640 1 -- 172.21.15.77:0/665786429 >> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] conn(0x7f4558072410 msgr2=0x7f4558074850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f457e334640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.121:6803/3650663429,v1:172.21.15.121:6805/3650663429] conn(0x7f4558072410 0x7f4558074850 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f457e334640 1 -- 172.21.15.77:0/665786429 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f455806be30 msgr2=0x7f455806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f457e334640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f455806be30 0x7f455806e2f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4560034000 tx=0x7f456002e000 comp rx=0 tx=0).stop 2024-04-25T01:24:20.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f457e334640 1 -- 172.21.15.77:0/665786429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45780ce470 msgr2=0x7f457814c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f457e334640 1 --2- 172.21.15.77:0/665786429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45780ce470 0x7f457814c000 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f456800d800 tx=0x7f456800dcc0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f457c8d1640 1 -- 172.21.15.77:0/665786429 reap_dead start 2024-04-25T01:24:20.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f457e334640 1 -- 172.21.15.77:0/665786429 shutdown_connections 2024-04-25T01:24:20.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f457e334640 1 -- 172.21.15.77:0/665786429 >> 172.21.15.77:0/665786429 conn(0x7f45780cfae0 msgr2=0x7f457806d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:20.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f457e334640 1 -- 172.21.15.77:0/665786429 shutdown_connections 2024-04-25T01:24:20.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.008+0000 7f457e334640 1 -- 172.21.15.77:0/665786429 wait complete. 2024-04-25T01:24:20.999 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-25T01:24:21.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.115+0000 7efd4c488640 1 Processor -- start 2024-04-25T01:24:21.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.115+0000 7efd4c488640 1 -- start start 2024-04-25T01:24:21.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.116+0000 7efd4c488640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efd4415e4c0 0x7efd440d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.116+0000 7efd4c488640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efd440d4a10 0x7efd440b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.116+0000 7efd4c488640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd440d1e90 con 0x7efd440d2d40 2024-04-25T01:24:21.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.116+0000 7efd4c488640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7efd440b54a0 con 0x7efd440d4a10 2024-04-25T01:24:21.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.116+0000 7efd4c488640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7efd440d2130 con 0x7efd4415e4c0 2024-04-25T01:24:21.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.117+0000 7efd49a23640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efd440d4a10 0x7efd440b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.117+0000 7efd4aa25640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efd440d2d40 0x7efd440d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38372/0 (socket says 172.21.15.77:38372) 2024-04-25T01:24:21.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.117+0000 7efd4aa25640 1 -- 172.21.15.77:0/3685115048 learned_addr learned my addr 172.21.15.77:0/3685115048 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:21.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.117+0000 7efd4a224640 1 --2- 172.21.15.77:0/3685115048 >> v2:172.21.15.121:3300/0 conn(0x7efd4415e4c0 0x7efd440d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.117+0000 7efd49a23640 1 -- 172.21.15.77:0/3685115048 >> v2:172.21.15.121:3300/0 conn(0x7efd4415e4c0 msgr2=0x7efd440d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:21.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.117+0000 7efd49a23640 1 --2- 172.21.15.77:0/3685115048 >> v2:172.21.15.121:3300/0 conn(0x7efd4415e4c0 0x7efd440d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.117+0000 7efd49a23640 1 -- 172.21.15.77:0/3685115048 >> v1:172.21.15.77:6789/0 conn(0x7efd440d2d40 legacy=0x7efd440d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.117+0000 7efd49a23640 1 -- 172.21.15.77:0/3685115048 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd440b5780 con 0x7efd440d4a10 2024-04-25T01:24:21.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.117+0000 7efd49222640 1 -- 172.21.15.77:0/3685115048 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2218375821 0 0) 0x7efd440d1e90 con 0x7efd440d2d40 2024-04-25T01:24:21.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.117+0000 7efd4a224640 1 --2- 172.21.15.77:0/3685115048 >> v2:172.21.15.121:3300/0 conn(0x7efd4415e4c0 0x7efd440d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:21.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.117+0000 7efd49a23640 1 --2- 172.21.15.77:0/3685115048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd440d4a10 0x7efd440b3ee0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7efd44067960 tx=0x7efd3802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b9c6d7ac76cc16 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.118+0000 7efd49222640 1 -- 172.21.15.77:0/3685115048 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38037070 con 0x7efd440d4a10 2024-04-25T01:24:21.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.118+0000 7efd49222640 1 -- 172.21.15.77:0/3685115048 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd38032440 con 0x7efd440d4a10 2024-04-25T01:24:21.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.118+0000 7efd49222640 1 -- 172.21.15.77:0/3685115048 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3802ae60 con 0x7efd440d4a10 2024-04-25T01:24:21.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.118+0000 7efd4c488640 1 -- 172.21.15.77:0/3685115048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd440d4a10 msgr2=0x7efd440b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.118+0000 7efd4c488640 1 --2- 172.21.15.77:0/3685115048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd440d4a10 0x7efd440b3ee0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7efd44067960 tx=0x7efd3802a110 comp rx=0 tx=0).stop 2024-04-25T01:24:21.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.118+0000 7efd4c488640 1 -- 172.21.15.77:0/3685115048 shutdown_connections 2024-04-25T01:24:21.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.118+0000 7efd4c488640 1 --2- 172.21.15.77:0/3685115048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd440d4a10 0x7efd440b3ee0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.118+0000 7efd4c488640 1 --2- 172.21.15.77:0/3685115048 >> v2:172.21.15.121:3300/0 conn(0x7efd4415e4c0 0x7efd440d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.118+0000 7efd4c488640 1 -- 172.21.15.77:0/3685115048 >> 172.21.15.77:0/3685115048 conn(0x7efd440d3ef0 msgr2=0x7efd440d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.118+0000 7efd4c488640 1 -- 172.21.15.77:0/3685115048 shutdown_connections 2024-04-25T01:24:21.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.118+0000 7efd4c488640 1 -- 172.21.15.77:0/3685115048 wait complete. 2024-04-25T01:24:21.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.119+0000 7efd4c488640 1 Processor -- start 2024-04-25T01:24:21.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.119+0000 7efd4c488640 1 -- start start 2024-04-25T01:24:21.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4c488640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd440d2d40 0x7efd441522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4c488640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd440d4a10 0x7efd440b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4c488640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd4415e4c0 0x7efd44160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4c488640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efd4415ea20 con 0x7efd4415e4c0 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4c488640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efd4415e8a0 con 0x7efd440d4a10 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4c488640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efd440d1010 con 0x7efd440d2d40 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4aa25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd4415e4c0 0x7efd44160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4aa25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd4415e4c0 0x7efd44160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49828/0 (socket says 172.21.15.77:49828) 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4aa25640 1 -- 172.21.15.77:0/4038489208 learned_addr learned my addr 172.21.15.77:0/4038489208 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4aa25640 1 -- 172.21.15.77:0/4038489208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd440d2d40 msgr2=0x7efd441522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4aa25640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd440d2d40 0x7efd441522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4aa25640 1 -- 172.21.15.77:0/4038489208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd440d4a10 msgr2=0x7efd440b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4aa25640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd440d4a10 0x7efd440b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.120+0000 7efd4aa25640 1 -- 172.21.15.77:0/4038489208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd44152820 con 0x7efd4415e4c0 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.121+0000 7efd4aa25640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd4415e4c0 0x7efd44160450 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7efd3400c450 tx=0x7efd3400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.121+0000 7efd42ffd640 1 -- 172.21.15.77:0/4038489208 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd34018070 con 0x7efd4415e4c0 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.121+0000 7efd42ffd640 1 -- 172.21.15.77:0/4038489208 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd34007500 con 0x7efd4415e4c0 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.121+0000 7efd4c488640 1 -- 172.21.15.77:0/4038489208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd440b8960 con 0x7efd4415e4c0 2024-04-25T01:24:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.121+0000 7efd42ffd640 1 -- 172.21.15.77:0/4038489208 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd34004a20 con 0x7efd4415e4c0 2024-04-25T01:24:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.121+0000 7efd4c488640 1 -- 172.21.15.77:0/4038489208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd440b8f20 con 0x7efd4415e4c0 2024-04-25T01:24:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.122+0000 7efd4c488640 1 -- 172.21.15.77:0/4038489208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd44068fb0 con 0x7efd4415e4c0 2024-04-25T01:24:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.122+0000 7efd42ffd640 1 -- 172.21.15.77:0/4038489208 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efd3401b070 con 0x7efd4415e4c0 2024-04-25T01:24:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.123+0000 7efd42ffd640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efd3006be30 0x7efd3006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.123+0000 7efd42ffd640 1 -- 172.21.15.77:0/4038489208 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7efd34089f90 con 0x7efd4415e4c0 2024-04-25T01:24:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.123+0000 7efd4a224640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efd3006be30 0x7efd3006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.123+0000 7efd4a224640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efd3006be30 0x7efd3006e2f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efd4415c450 tx=0x7efd3c008330 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.127+0000 7efd42ffd640 1 -- 172.21.15.77:0/4038489208 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7efd34058a90 con 0x7efd4415e4c0 2024-04-25T01:24:21.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.304+0000 7efd4c488640 1 -- 172.21.15.77:0/4038489208 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7efd44073c20 con 0x7efd3006be30 2024-04-25T01:24:21.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.304+0000 7efd42ffd640 1 -- 172.21.15.77:0/4038489208 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7efd44073c20 con 0x7efd3006be30 2024-04-25T01:24:21.285 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 4 to deep-scrub 2024-04-25T01:24:21.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 -- 172.21.15.77:0/4038489208 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efd3006be30 msgr2=0x7efd3006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efd3006be30 0x7efd3006e2f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efd4415c450 tx=0x7efd3c008330 comp rx=0 tx=0).stop 2024-04-25T01:24:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 -- 172.21.15.77:0/4038489208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd4415e4c0 msgr2=0x7efd44160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd4415e4c0 0x7efd44160450 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7efd3400c450 tx=0x7efd3400c910 comp rx=0 tx=0).stop 2024-04-25T01:24:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 -- 172.21.15.77:0/4038489208 shutdown_connections 2024-04-25T01:24:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7efd3006be30 0x7efd3006e2f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd4415e4c0 0x7efd44160450 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd440d4a10 0x7efd440b7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 --2- 172.21.15.77:0/4038489208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd440d2d40 0x7efd441522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 -- 172.21.15.77:0/4038489208 >> 172.21.15.77:0/4038489208 conn(0x7efd440d3ef0 msgr2=0x7efd440d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 -- 172.21.15.77:0/4038489208 shutdown_connections 2024-04-25T01:24:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.306+0000 7efd4c488640 1 -- 172.21.15.77:0/4038489208 wait complete. 2024-04-25T01:24:21.297 INFO:tasks.ceph:Scrubbing osd.5 2024-04-25T01:24:21.298 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:24:21.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.417+0000 7f60a9064640 1 Processor -- start 2024-04-25T01:24:21.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.417+0000 7f60a9064640 1 -- start start 2024-04-25T01:24:21.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.418+0000 7f60a9064640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f60a415e600 0x7f60a4165280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.418+0000 7f60a9064640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60a406e080 con 0x7f60a4154b00 2024-04-25T01:24:21.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.418+0000 7f60a9064640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60a406e2f0 con 0x7f60a406dbf0 2024-04-25T01:24:21.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.418+0000 7f60a9064640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f60a406e5d0 con 0x7f60a415e600 2024-04-25T01:24:21.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.418+0000 7f60a359e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f60a415e600 0x7f60a4165280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.418+0000 7f60a2d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f60a4154b00 0x7f60a40ca1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38388/0 (socket says 172.21.15.77:38388) 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.418+0000 7f60a2d9d640 1 -- 172.21.15.77:0/1147697399 learned_addr learned my addr 172.21.15.77:0/1147697399 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.418+0000 7f60a359e640 1 -- 172.21.15.77:0/1147697399 >> v1:172.21.15.121:6789/0 conn(0x7f60a406dbf0 legacy=0x7f60a415def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.418+0000 7f60a359e640 1 -- 172.21.15.77:0/1147697399 >> v1:172.21.15.77:6789/0 conn(0x7f60a4154b00 legacy=0x7f60a40ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.418+0000 7f60a359e640 1 -- 172.21.15.77:0/1147697399 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a406e8b0 con 0x7f60a415e600 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.418+0000 7f60a1d9b640 1 -- 172.21.15.77:0/1147697399 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2134816293 0 0) 0x7f60a406e080 con 0x7f60a4154b00 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.419+0000 7f60a359e640 1 --2- 172.21.15.77:0/1147697399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60a415e600 0x7f60a4165280 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f609c009d90 tx=0x7f609c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1483b609569e33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.419+0000 7f60a1d9b640 1 -- 172.21.15.77:0/1147697399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c007740 con 0x7f60a415e600 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.419+0000 7f60a1d9b640 1 -- 172.21.15.77:0/1147697399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f609c0078e0 con 0x7f60a415e600 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.419+0000 7f60a1d9b640 1 -- 172.21.15.77:0/1147697399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c007cb0 con 0x7f60a415e600 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.419+0000 7f60a9064640 1 -- 172.21.15.77:0/1147697399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60a415e600 msgr2=0x7f60a4165280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.419+0000 7f60a9064640 1 --2- 172.21.15.77:0/1147697399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60a415e600 0x7f60a4165280 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f609c009d90 tx=0x7f609c02a660 comp rx=0 tx=0).stop 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.419+0000 7f60a9064640 1 -- 172.21.15.77:0/1147697399 shutdown_connections 2024-04-25T01:24:21.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.419+0000 7f60a9064640 1 --2- 172.21.15.77:0/1147697399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60a415e600 0x7f60a4165280 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.419+0000 7f60a9064640 1 -- 172.21.15.77:0/1147697399 >> 172.21.15.77:0/1147697399 conn(0x7f60a40cfae0 msgr2=0x7f60a40cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.419+0000 7f60a9064640 1 -- 172.21.15.77:0/1147697399 shutdown_connections 2024-04-25T01:24:21.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.419+0000 7f60a9064640 1 -- 172.21.15.77:0/1147697399 wait complete. 2024-04-25T01:24:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.421+0000 7f60a9064640 1 Processor -- start 2024-04-25T01:24:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.421+0000 7f60a9064640 1 -- start start 2024-04-25T01:24:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a9064640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60a406dbf0 0x7f60a413f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a9064640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60a4154b00 0x7f60a4140a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a9064640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60a415e600 0x7f60a40b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a9064640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f60a40cf2e0 con 0x7f60a415e600 2024-04-25T01:24:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a9064640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f60a40cf160 con 0x7f60a406dbf0 2024-04-25T01:24:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a9064640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f60a40cf460 con 0x7f60a4154b00 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60a415e600 0x7f60a40b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60a415e600 0x7f60a40b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49858/0 (socket says 172.21.15.77:49858) 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a359e640 1 -- 172.21.15.77:0/2428648150 learned_addr learned my addr 172.21.15.77:0/2428648150 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a2d9d640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60a406dbf0 0x7f60a413f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a359e640 1 -- 172.21.15.77:0/2428648150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60a4154b00 msgr2=0x7f60a4140a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a359e640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60a4154b00 0x7f60a4140a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a359e640 1 -- 172.21.15.77:0/2428648150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60a406dbf0 msgr2=0x7f60a413f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a359e640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60a406dbf0 0x7f60a413f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a359e640 1 -- 172.21.15.77:0/2428648150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a41f6150 con 0x7f60a415e600 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a359e640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60a415e600 0x7f60a40b3ed0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f609c009eb0 tx=0x7f609c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f608b7fe640 1 -- 172.21.15.77:0/2428648150 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c03f070 con 0x7f60a415e600 2024-04-25T01:24:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f608b7fe640 1 -- 172.21.15.77:0/2428648150 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f609c031c90 con 0x7f60a415e600 2024-04-25T01:24:21.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a9064640 1 -- 172.21.15.77:0/2428648150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60a413ff50 con 0x7f60a415e600 2024-04-25T01:24:21.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f608b7fe640 1 -- 172.21.15.77:0/2428648150 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c043630 con 0x7f60a415e600 2024-04-25T01:24:21.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.422+0000 7f60a9064640 1 -- 172.21.15.77:0/2428648150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60a4140510 con 0x7f60a415e600 2024-04-25T01:24:21.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.423+0000 7f60a9064640 1 -- 172.21.15.77:0/2428648150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f60a413fda0 con 0x7f60a415e600 2024-04-25T01:24:21.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.424+0000 7f608b7fe640 1 -- 172.21.15.77:0/2428648150 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f609c03b410 con 0x7f60a415e600 2024-04-25T01:24:21.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.424+0000 7f608b7fe640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f609006be30 0x7f609006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.424+0000 7f608b7fe640 1 -- 172.21.15.77:0/2428648150 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f609c0ac530 con 0x7f60a415e600 2024-04-25T01:24:21.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.424+0000 7f60a2d9d640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f609006be30 0x7f609006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.424+0000 7f608b7fe640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] conn(0x7f6090072410 0x7f6090074850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.424+0000 7f608b7fe640 1 -- 172.21.15.77:0/2428648150 --> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f609c006b90 con 0x7f6090072410 2024-04-25T01:24:21.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.424+0000 7f608b7fe640 1 -- 172.21.15.77:0/2428648150 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f609c0ac9c0 con 0x7f60a415e600 2024-04-25T01:24:21.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.425+0000 7f60a2d9d640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f609006be30 0x7f609006e2f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6094007fd0 tx=0x7f6094009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.425+0000 7f60a259c640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] conn(0x7f6090072410 0x7f6090074850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.425+0000 7f60a259c640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] conn(0x7f6090072410 0x7f6090074850 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.426+0000 7f608b7fe640 1 -- 172.21.15.77:0/2428648150 <== osd.5 v2:172.21.15.121:6817/3849358101 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f609c006b90 con 0x7f6090072410 2024-04-25T01:24:21.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.439+0000 7f60a9064640 1 -- 172.21.15.77:0/2428648150 --> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f60a4068fb0 con 0x7f6090072410 2024-04-25T01:24:21.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.439+0000 7f608b7fe640 1 -- 172.21.15.77:0/2428648150 <== osd.5 v2:172.21.15.121:6817/3849358101 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f60a4068fb0 con 0x7f6090072410 2024-04-25T01:24:21.420 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:24:21.420 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:24:21.421 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:24:21.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.440+0000 7f60a9064640 1 -- 172.21.15.77:0/2428648150 >> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] conn(0x7f6090072410 msgr2=0x7f6090074850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.440+0000 7f60a9064640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.121:6817/3849358101,v1:172.21.15.121:6819/3849358101] conn(0x7f6090072410 0x7f6090074850 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.440+0000 7f60a9064640 1 -- 172.21.15.77:0/2428648150 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f609006be30 msgr2=0x7f609006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.440+0000 7f60a9064640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f609006be30 0x7f609006e2f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6094007fd0 tx=0x7f6094009040 comp rx=0 tx=0).stop 2024-04-25T01:24:21.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.440+0000 7f60a9064640 1 -- 172.21.15.77:0/2428648150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60a415e600 msgr2=0x7f60a40b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.440+0000 7f60a9064640 1 --2- 172.21.15.77:0/2428648150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60a415e600 0x7f60a40b3ed0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f609c009eb0 tx=0x7f609c004a40 comp rx=0 tx=0).stop 2024-04-25T01:24:21.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.440+0000 7f60a359e640 1 -- 172.21.15.77:0/2428648150 reap_dead start 2024-04-25T01:24:21.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.440+0000 7f60a9064640 1 -- 172.21.15.77:0/2428648150 shutdown_connections 2024-04-25T01:24:21.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.440+0000 7f60a9064640 1 -- 172.21.15.77:0/2428648150 >> 172.21.15.77:0/2428648150 conn(0x7f60a40cfae0 msgr2=0x7f60a4143810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.440+0000 7f60a9064640 1 -- 172.21.15.77:0/2428648150 shutdown_connections 2024-04-25T01:24:21.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.440+0000 7f60a9064640 1 -- 172.21.15.77:0/2428648150 wait complete. 2024-04-25T01:24:21.431 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-25T01:24:21.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.550+0000 7f646b929640 1 Processor -- start 2024-04-25T01:24:21.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.550+0000 7f646b929640 1 -- start start 2024-04-25T01:24:21.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f646b929640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6464145590 0x7f64640b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f646b929640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64640b4240 con 0x7f646406e280 2024-04-25T01:24:21.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f646b929640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64640b4f60 con 0x7f646415f9b0 2024-04-25T01:24:21.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f646b929640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6464160830 con 0x7f6464145590 2024-04-25T01:24:21.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f6469ec6640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f646406e280 0x7f64640ce820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38394/0 (socket says 172.21.15.77:38394) 2024-04-25T01:24:21.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f6469ec6640 1 -- 172.21.15.77:0/2923554607 learned_addr learned my addr 172.21.15.77:0/2923554607 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:21.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1053304975 0 0) 0x7f64640b4240 con 0x7f646406e280 2024-04-25T01:24:21.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6450003520 con 0x7f646406e280 2024-04-25T01:24:21.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f64696c5640 1 --2- 172.21.15.77:0/2923554607 >> v2:172.21.15.121:3300/0 conn(0x7f6464145590 0x7f64640b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1584217791 0 0) 0x7f6450003520 con 0x7f646406e280 2024-04-25T01:24:21.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 >> v2:172.21.15.121:3300/0 conn(0x7f6464145590 msgr2=0x7f64640b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:21.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 --2- 172.21.15.77:0/2923554607 >> v2:172.21.15.121:3300/0 conn(0x7f6464145590 0x7f64640b4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 >> v1:172.21.15.121:6789/0 conn(0x7f646415f9b0 legacy=0x7f64640ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64640b39a0 con 0x7f646406e280 2024-04-25T01:24:21.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7f645c0019f0 con 0x7f646406e280 2024-04-25T01:24:21.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 >> v1:172.21.15.77:6789/0 conn(0x7f646406e280 legacy=0x7f64640ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 --2- 172.21.15.77:0/2923554607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64500039b0 0x7f6450003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 --2- 172.21.15.77:0/2923554607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6450004800 0x7f6450004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.551+0000 7f645bfff640 1 --2- 172.21.15.77:0/2923554607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64500051c0 0x7f645000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f645000c030 con 0x7f6450004800 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f645000c1e0 con 0x7f64500051c0 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f645000c3e0 con 0x7f64500039b0 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f6468ec4640 1 --2- 172.21.15.77:0/2923554607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6450004800 0x7f6450004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f64696c5640 1 --2- 172.21.15.77:0/2923554607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64500039b0 0x7f6450003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f6469ec6640 1 --2- 172.21.15.77:0/2923554607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64500051c0 0x7f645000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f6468ec4640 1 -- 172.21.15.77:0/2923554607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64500039b0 msgr2=0x7f6450003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f6468ec4640 1 --2- 172.21.15.77:0/2923554607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64500039b0 0x7f6450003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f6468ec4640 1 -- 172.21.15.77:0/2923554607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64500051c0 msgr2=0x7f645000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f6468ec4640 1 --2- 172.21.15.77:0/2923554607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64500051c0 0x7f645000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f6469ec6640 1 -- 172.21.15.77:0/2923554607 reap_dead start 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.552+0000 7f6468ec4640 1 -- 172.21.15.77:0/2923554607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6450003520 con 0x7f6450004800 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.553+0000 7f6468ec4640 1 --2- 172.21.15.77:0/2923554607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6450004800 0x7f6450004c80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f6454001750 tx=0x7f6450005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0428d46a18b3f94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.553+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6454001370 con 0x7f6450004800 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.553+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64540052d0 con 0x7f6450004800 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.553+0000 7f645bfff640 1 -- 172.21.15.77:0/2923554607 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645402abc0 con 0x7f6450004800 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.553+0000 7f646b929640 1 -- 172.21.15.77:0/2923554607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6450004800 msgr2=0x7f6450004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.553+0000 7f646b929640 1 --2- 172.21.15.77:0/2923554607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6450004800 0x7f6450004c80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f6454001750 tx=0x7f6450005830 comp rx=0 tx=0).stop 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.553+0000 7f646b929640 1 -- 172.21.15.77:0/2923554607 shutdown_connections 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.553+0000 7f646b929640 1 --2- 172.21.15.77:0/2923554607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6450004800 0x7f6450004c80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.553+0000 7f646b929640 1 -- 172.21.15.77:0/2923554607 >> 172.21.15.77:0/2923554607 conn(0x7f64640b1820 msgr2=0x7f64640b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.553+0000 7f646b929640 1 -- 172.21.15.77:0/2923554607 shutdown_connections 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.553+0000 7f646b929640 1 -- 172.21.15.77:0/2923554607 wait complete. 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.554+0000 7f646b929640 1 Processor -- start 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.554+0000 7f646b929640 1 -- start start 2024-04-25T01:24:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f646b929640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6450004800 0x7f646415b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f646b929640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f646415df00 0x7f646415e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f646b929640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64641e61b0 0x7f64641e6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f646b929640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f64640b3f60 con 0x7f64641e61b0 2024-04-25T01:24:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f646b929640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f646406cde0 con 0x7f6450004800 2024-04-25T01:24:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f646b929640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6464145970 con 0x7f646415df00 2024-04-25T01:24:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f6469ec6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64641e61b0 0x7f64641e6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f6469ec6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64641e61b0 0x7f64641e6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49872/0 (socket says 172.21.15.77:49872) 2024-04-25T01:24:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f6469ec6640 1 -- 172.21.15.77:0/1095630648 learned_addr learned my addr 172.21.15.77:0/1095630648 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f6468ec4640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f646415df00 0x7f646415e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f64696c5640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6450004800 0x7f646415b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f6469ec6640 1 -- 172.21.15.77:0/1095630648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f646415df00 msgr2=0x7f646415e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f6469ec6640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f646415df00 0x7f646415e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f6469ec6640 1 -- 172.21.15.77:0/1095630648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6450004800 msgr2=0x7f646415b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f6469ec6640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6450004800 0x7f646415b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f6469ec6640 1 -- 172.21.15.77:0/1095630648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64641e6ee0 con 0x7f64641e61b0 2024-04-25T01:24:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f64696c5640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6450004800 0x7f646415b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.555+0000 7f6468ec4640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f646415df00 0x7f646415e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.556+0000 7f6469ec6640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64641e61b0 0x7f64641e6630 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f645c000f80 tx=0x7f645c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.556+0000 7f6459ffb640 1 -- 172.21.15.77:0/1095630648 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c00abc0 con 0x7f64641e61b0 2024-04-25T01:24:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.556+0000 7f6459ffb640 1 -- 172.21.15.77:0/1095630648 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f645c019070 con 0x7f64641e61b0 2024-04-25T01:24:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.556+0000 7f6459ffb640 1 -- 172.21.15.77:0/1095630648 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c0133f0 con 0x7f64641e61b0 2024-04-25T01:24:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.556+0000 7f646b929640 1 -- 172.21.15.77:0/1095630648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6464137ad0 con 0x7f64641e61b0 2024-04-25T01:24:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.556+0000 7f646b929640 1 -- 172.21.15.77:0/1095630648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6464138090 con 0x7f64641e61b0 2024-04-25T01:24:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.557+0000 7f646b929640 1 -- 172.21.15.77:0/1095630648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6464068fb0 con 0x7f64641e61b0 2024-04-25T01:24:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.557+0000 7f6459ffb640 1 -- 172.21.15.77:0/1095630648 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f645c013590 con 0x7f64641e61b0 2024-04-25T01:24:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.558+0000 7f6459ffb640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f644806be60 0x7f644806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.558+0000 7f6459ffb640 1 -- 172.21.15.77:0/1095630648 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f645c08b270 con 0x7f64641e61b0 2024-04-25T01:24:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.558+0000 7f64696c5640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f644806be60 0x7f644806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.558+0000 7f64696c5640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f644806be60 0x7f644806e320 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f64641413b0 tx=0x7f6460008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.562+0000 7f6459ffb640 1 -- 172.21.15.77:0/1095630648 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f645c059ce0 con 0x7f64641e61b0 2024-04-25T01:24:21.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.734+0000 7f646b929640 1 -- 172.21.15.77:0/1095630648 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f64640686a0 con 0x7f644806be60 2024-04-25T01:24:21.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.734+0000 7f6459ffb640 1 -- 172.21.15.77:0/1095630648 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6460014b50 con 0x7f644806be60 2024-04-25T01:24:21.715 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 5 to deep-scrub 2024-04-25T01:24:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 -- 172.21.15.77:0/1095630648 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f644806be60 msgr2=0x7f644806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f644806be60 0x7f644806e320 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f64641413b0 tx=0x7f6460008040 comp rx=0 tx=0).stop 2024-04-25T01:24:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 -- 172.21.15.77:0/1095630648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64641e61b0 msgr2=0x7f64641e6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64641e61b0 0x7f64641e6630 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f645c000f80 tx=0x7f645c007a90 comp rx=0 tx=0).stop 2024-04-25T01:24:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 -- 172.21.15.77:0/1095630648 shutdown_connections 2024-04-25T01:24:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f644806be60 0x7f644806e320 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64641e61b0 0x7f64641e6630 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f646415df00 0x7f646415e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 --2- 172.21.15.77:0/1095630648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6450004800 0x7f646415b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 -- 172.21.15.77:0/1095630648 >> 172.21.15.77:0/1095630648 conn(0x7f64640b1820 msgr2=0x7f64640b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 -- 172.21.15.77:0/1095630648 shutdown_connections 2024-04-25T01:24:21.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.736+0000 7f646b929640 1 -- 172.21.15.77:0/1095630648 wait complete. 2024-04-25T01:24:21.727 INFO:tasks.ceph:Scrubbing osd.6 2024-04-25T01:24:21.727 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:24:21.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.846+0000 7f81240d2640 1 Processor -- start 2024-04-25T01:24:21.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.846+0000 7f81240d2640 1 -- start start 2024-04-25T01:24:21.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.847+0000 7f81240d2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f811c06cb40 0x7f811c070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.847+0000 7f81240d2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f811c0722c0 con 0x7f811c071110 2024-04-25T01:24:21.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.847+0000 7f81240d2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f811c0ca1d0 con 0x7f811c15f870 2024-04-25T01:24:21.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.847+0000 7f81240d2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f811c0725a0 con 0x7f811c06cb40 2024-04-25T01:24:21.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.847+0000 7f812166d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f811c06cb40 0x7f811c070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.847+0000 7f812266f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f811c071110 0x7f811c0715a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38406/0 (socket says 172.21.15.77:38406) 2024-04-25T01:24:21.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.847+0000 7f812266f640 1 -- 172.21.15.77:0/3744762156 learned_addr learned my addr 172.21.15.77:0/3744762156 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:21.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.847+0000 7f812166d640 1 -- 172.21.15.77:0/3744762156 >> v1:172.21.15.121:6789/0 conn(0x7f811c15f870 legacy=0x7f811c15fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:21.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.847+0000 7f812166d640 1 -- 172.21.15.77:0/3744762156 >> v1:172.21.15.77:6789/0 conn(0x7f811c071110 legacy=0x7f811c0715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.848+0000 7f812166d640 1 -- 172.21.15.77:0/3744762156 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811c0ca4b0 con 0x7f811c06cb40 2024-04-25T01:24:21.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.848+0000 7f812166d640 1 --2- 172.21.15.77:0/3744762156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f811c06cb40 0x7f811c070bd0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f810c0097d0 tx=0x7f810c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b40c25cc42e85f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.848+0000 7f8120e6c640 1 -- 172.21.15.77:0/3744762156 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c037070 con 0x7f811c06cb40 2024-04-25T01:24:21.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.848+0000 7f8120e6c640 1 -- 172.21.15.77:0/3744762156 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f810c032440 con 0x7f811c06cb40 2024-04-25T01:24:21.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.848+0000 7f8120e6c640 1 -- 172.21.15.77:0/3744762156 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c032810 con 0x7f811c06cb40 2024-04-25T01:24:21.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.849+0000 7f81240d2640 1 -- 172.21.15.77:0/3744762156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f811c06cb40 msgr2=0x7f811c070bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.849+0000 7f81240d2640 1 --2- 172.21.15.77:0/3744762156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f811c06cb40 0x7f811c070bd0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f810c0097d0 tx=0x7f810c02a210 comp rx=0 tx=0).stop 2024-04-25T01:24:21.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.849+0000 7f81240d2640 1 -- 172.21.15.77:0/3744762156 shutdown_connections 2024-04-25T01:24:21.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.849+0000 7f81240d2640 1 --2- 172.21.15.77:0/3744762156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f811c06cb40 0x7f811c070bd0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.849+0000 7f81240d2640 1 -- 172.21.15.77:0/3744762156 >> 172.21.15.77:0/3744762156 conn(0x7f811c0cfae0 msgr2=0x7f811c0cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.849+0000 7f81240d2640 1 -- 172.21.15.77:0/3744762156 shutdown_connections 2024-04-25T01:24:21.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.849+0000 7f81240d2640 1 -- 172.21.15.77:0/3744762156 wait complete. 2024-04-25T01:24:21.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.850+0000 7f81240d2640 1 Processor -- start 2024-04-25T01:24:21.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.850+0000 7f81240d2640 1 -- start start 2024-04-25T01:24:21.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f81240d2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f811c06cb40 0x7f811c1f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f81240d2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f811c071110 0x7f811c0b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f81240d2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f811c15f870 0x7f811c0c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f81240d2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f811c06cfa0 con 0x7f811c071110 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f81240d2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f811c145910 con 0x7f811c06cb40 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f81240d2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f811c1f8540 con 0x7f811c15f870 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f812166d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f811c071110 0x7f811c0b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f812166d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f811c071110 0x7f811c0b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49886/0 (socket says 172.21.15.77:49886) 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f812166d640 1 -- 172.21.15.77:0/2629705020 learned_addr learned my addr 172.21.15.77:0/2629705020 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f812166d640 1 -- 172.21.15.77:0/2629705020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f811c15f870 msgr2=0x7f811c0c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f812166d640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f811c15f870 0x7f811c0c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f812166d640 1 -- 172.21.15.77:0/2629705020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f811c06cb40 msgr2=0x7f811c1f7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f812166d640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f811c06cb40 0x7f811c1f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f812166d640 1 -- 172.21.15.77:0/2629705020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811c1f7ca0 con 0x7f811c071110 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f812166d640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f811c071110 0x7f811c0b4510 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f810c000c00 tx=0x7f810c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f81127fc640 1 -- 172.21.15.77:0/2629705020 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c037040 con 0x7f811c071110 2024-04-25T01:24:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f81127fc640 1 -- 172.21.15.77:0/2629705020 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f810c004230 con 0x7f811c071110 2024-04-25T01:24:21.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.851+0000 7f81240d2640 1 -- 172.21.15.77:0/2629705020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811c1f7f80 con 0x7f811c071110 2024-04-25T01:24:21.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.852+0000 7f81127fc640 1 -- 172.21.15.77:0/2629705020 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c032e20 con 0x7f811c071110 2024-04-25T01:24:21.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.852+0000 7f81240d2640 1 -- 172.21.15.77:0/2629705020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f811c1f4230 con 0x7f811c071110 2024-04-25T01:24:21.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.852+0000 7f81240d2640 1 -- 172.21.15.77:0/2629705020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f811c15c010 con 0x7f811c071110 2024-04-25T01:24:21.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.853+0000 7f81127fc640 1 -- 172.21.15.77:0/2629705020 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f810c0306f0 con 0x7f811c071110 2024-04-25T01:24:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.853+0000 7f81127fc640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f810006be30 0x7f810006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.853+0000 7f81127fc640 1 -- 172.21.15.77:0/2629705020 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f810c0abc60 con 0x7f811c071110 2024-04-25T01:24:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.853+0000 7f8121e6e640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f810006be30 0x7f810006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.853+0000 7f81127fc640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] conn(0x7f8100072410 0x7f8100074850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.853+0000 7f81127fc640 1 -- 172.21.15.77:0/2629705020 --> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f810c045e80 con 0x7f8100072410 2024-04-25T01:24:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.853+0000 7f81127fc640 1 -- 172.21.15.77:0/2629705020 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f810c0b1070 con 0x7f811c071110 2024-04-25T01:24:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.854+0000 7f8121e6e640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f810006be30 0x7f810006e2f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f811800ab90 tx=0x7f8118009230 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.854+0000 7f812266f640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] conn(0x7f8100072410 0x7f8100074850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.854+0000 7f812266f640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] conn(0x7f8100072410 0x7f8100074850 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.855+0000 7f81127fc640 1 -- 172.21.15.77:0/2629705020 <== osd.6 v2:172.21.15.121:6816/3890411649 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f810c045e80 con 0x7f8100072410 2024-04-25T01:24:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.868+0000 7f81240d2640 1 -- 172.21.15.77:0/2629705020 --> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f811c0b4b10 con 0x7f8100072410 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.868+0000 7f81127fc640 1 -- 172.21.15.77:0/2629705020 <== osd.6 v2:172.21.15.121:6816/3890411649 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f811c0b4b10 con 0x7f8100072410 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:24:21.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.869+0000 7f81240d2640 1 -- 172.21.15.77:0/2629705020 >> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] conn(0x7f8100072410 msgr2=0x7f8100074850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.869+0000 7f81240d2640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.121:6816/3890411649,v1:172.21.15.121:6818/3890411649] conn(0x7f8100072410 0x7f8100074850 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.869+0000 7f81240d2640 1 -- 172.21.15.77:0/2629705020 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f810006be30 msgr2=0x7f810006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.869+0000 7f81240d2640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f810006be30 0x7f810006e2f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f811800ab90 tx=0x7f8118009230 comp rx=0 tx=0).stop 2024-04-25T01:24:21.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.869+0000 7f81240d2640 1 -- 172.21.15.77:0/2629705020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f811c071110 msgr2=0x7f811c0b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.869+0000 7f81240d2640 1 --2- 172.21.15.77:0/2629705020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f811c071110 0x7f811c0b4510 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f810c000c00 tx=0x7f810c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.869+0000 7f812266f640 1 -- 172.21.15.77:0/2629705020 reap_dead start 2024-04-25T01:24:21.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.869+0000 7f81240d2640 1 -- 172.21.15.77:0/2629705020 shutdown_connections 2024-04-25T01:24:21.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.869+0000 7f81240d2640 1 -- 172.21.15.77:0/2629705020 >> 172.21.15.77:0/2629705020 conn(0x7f811c0cfae0 msgr2=0x7f811c1401e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.869+0000 7f81240d2640 1 -- 172.21.15.77:0/2629705020 shutdown_connections 2024-04-25T01:24:21.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.869+0000 7f81240d2640 1 -- 172.21.15.77:0/2629705020 wait complete. 2024-04-25T01:24:21.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-25T01:24:21.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.976+0000 7facf7777640 1 Processor -- start 2024-04-25T01:24:21.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.976+0000 7facf7777640 1 -- start start 2024-04-25T01:24:21.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.977+0000 7facf7777640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7facf00cff30 0x7facf00d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.977+0000 7facf7777640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7facf006d240 0x7facf00aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.977+0000 7facf7777640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facf00d0850 con 0x7facf015e420 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.977+0000 7facf7777640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7facf006d740 con 0x7facf00cff30 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.977+0000 7facf7777640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7facf006da20 con 0x7facf006d240 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.977+0000 7facf5513640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7facf00cff30 0x7facf00d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.977+0000 7facf4d12640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7facf015e420 0x7facf00d0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38410/0 (socket says 172.21.15.77:38410) 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.977+0000 7facf4d12640 1 -- 172.21.15.77:0/3356846823 learned_addr learned my addr 172.21.15.77:0/3356846823 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.978+0000 7facf5d14640 1 --2- 172.21.15.77:0/3356846823 >> v2:172.21.15.121:3300/0 conn(0x7facf006d240 0x7facf00aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.978+0000 7facf5513640 1 -- 172.21.15.77:0/3356846823 >> v2:172.21.15.121:3300/0 conn(0x7facf006d240 msgr2=0x7facf00aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.978+0000 7facf5513640 1 --2- 172.21.15.77:0/3356846823 >> v2:172.21.15.121:3300/0 conn(0x7facf006d240 0x7facf00aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.978+0000 7facf5513640 1 -- 172.21.15.77:0/3356846823 >> v1:172.21.15.77:6789/0 conn(0x7facf015e420 legacy=0x7facf00d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.978+0000 7face7fff640 1 -- 172.21.15.77:0/3356846823 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3882706663 0 0) 0x7facf00d0850 con 0x7facf015e420 2024-04-25T01:24:21.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.978+0000 7facf5513640 1 -- 172.21.15.77:0/3356846823 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf006dd00 con 0x7facf00cff30 2024-04-25T01:24:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.978+0000 7facf5513640 1 --2- 172.21.15.77:0/3356846823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf00cff30 0x7facf00d0310 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7facf00679a0 tx=0x7facec02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb44ac30261c34a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.978+0000 7face7fff640 1 -- 172.21.15.77:0/3356846823 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec037070 con 0x7facf00cff30 2024-04-25T01:24:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.979+0000 7face7fff640 1 -- 172.21.15.77:0/3356846823 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec032440 con 0x7facf00cff30 2024-04-25T01:24:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.979+0000 7face7fff640 1 -- 172.21.15.77:0/3356846823 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec032810 con 0x7facf00cff30 2024-04-25T01:24:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.979+0000 7facf7777640 1 -- 172.21.15.77:0/3356846823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf00cff30 msgr2=0x7facf00d0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.979+0000 7facf7777640 1 --2- 172.21.15.77:0/3356846823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf00cff30 0x7facf00d0310 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7facf00679a0 tx=0x7facec02a1a0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.979+0000 7facf7777640 1 -- 172.21.15.77:0/3356846823 shutdown_connections 2024-04-25T01:24:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.979+0000 7facf7777640 1 --2- 172.21.15.77:0/3356846823 >> v2:172.21.15.121:3300/0 conn(0x7facf006d240 0x7facf00aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.979+0000 7facf7777640 1 --2- 172.21.15.77:0/3356846823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf00cff30 0x7facf00d0310 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.979+0000 7facf7777640 1 -- 172.21.15.77:0/3356846823 >> 172.21.15.77:0/3356846823 conn(0x7facf00d1ca0 msgr2=0x7facf00d20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.979+0000 7facf7777640 1 -- 172.21.15.77:0/3356846823 shutdown_connections 2024-04-25T01:24:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.979+0000 7facf7777640 1 -- 172.21.15.77:0/3356846823 wait complete. 2024-04-25T01:24:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.980+0000 7facf7777640 1 Processor -- start 2024-04-25T01:24:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.980+0000 7facf7777640 1 -- start start 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf7777640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf006d240 0x7facf014a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf7777640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facf00cff30 0x7facf014ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf7777640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7facf015e420 0x7facf00b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf7777640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7facf00af490 con 0x7facf006d240 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf7777640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7facf00af310 con 0x7facf015e420 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf7777640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7facf00af610 con 0x7facf00cff30 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf5513640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf006d240 0x7facf014a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf5513640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf006d240 0x7facf014a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49910/0 (socket says 172.21.15.77:49910) 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf5513640 1 -- 172.21.15.77:0/2116286532 learned_addr learned my addr 172.21.15.77:0/2116286532 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf5513640 1 -- 172.21.15.77:0/2116286532 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facf00cff30 msgr2=0x7facf014ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf5513640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facf00cff30 0x7facf014ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf5d14640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7facf015e420 0x7facf00b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf5513640 1 -- 172.21.15.77:0/2116286532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7facf015e420 msgr2=0x7facf00b1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf5513640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7facf015e420 0x7facf00b1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.981+0000 7facf5513640 1 -- 172.21.15.77:0/2116286532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf014a820 con 0x7facf006d240 2024-04-25T01:24:21.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.982+0000 7facf5513640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf006d240 0x7facf014a2e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7facec000c00 tx=0x7facec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.982+0000 7face5ffb640 1 -- 172.21.15.77:0/2116286532 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec037040 con 0x7facf006d240 2024-04-25T01:24:21.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.982+0000 7face5ffb640 1 -- 172.21.15.77:0/2116286532 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec042070 con 0x7facf006d240 2024-04-25T01:24:21.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.982+0000 7facf7777640 1 -- 172.21.15.77:0/2116286532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf009de20 con 0x7facf006d240 2024-04-25T01:24:21.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.982+0000 7face5ffb640 1 -- 172.21.15.77:0/2116286532 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec030bd0 con 0x7facf006d240 2024-04-25T01:24:21.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.982+0000 7facf7777640 1 -- 172.21.15.77:0/2116286532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf009e3e0 con 0x7facf006d240 2024-04-25T01:24:21.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.983+0000 7facf7777640 1 -- 172.21.15.77:0/2116286532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf009eaf0 con 0x7facf006d240 2024-04-25T01:24:21.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.983+0000 7face5ffb640 1 -- 172.21.15.77:0/2116286532 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7facec047460 con 0x7facf006d240 2024-04-25T01:24:21.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.984+0000 7face5ffb640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facd406be60 0x7facd406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.984+0000 7face5ffb640 1 -- 172.21.15.77:0/2116286532 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7facec0ab960 con 0x7facf006d240 2024-04-25T01:24:21.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.984+0000 7facf4d12640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facd406be60 0x7facd406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.984+0000 7facf4d12640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facd406be60 0x7facd406e320 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7facf00c8820 tx=0x7face0007a00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:21.988+0000 7face5ffb640 1 -- 172.21.15.77:0/2116286532 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7facec032e20 con 0x7facf006d240 2024-04-25T01:24:22.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.160+0000 7facf7777640 1 -- 172.21.15.77:0/2116286532 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7facf0068fb0 con 0x7facd406be60 2024-04-25T01:24:22.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.160+0000 7face5ffb640 1 -- 172.21.15.77:0/2116286532 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7facf0068fb0 con 0x7facd406be60 2024-04-25T01:24:22.141 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 6 to deep-scrub 2024-04-25T01:24:22.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 -- 172.21.15.77:0/2116286532 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facd406be60 msgr2=0x7facd406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facd406be60 0x7facd406e320 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7facf00c8820 tx=0x7face0007a00 comp rx=0 tx=0).stop 2024-04-25T01:24:22.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 -- 172.21.15.77:0/2116286532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf006d240 msgr2=0x7facf014a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf006d240 0x7facf014a2e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7facec000c00 tx=0x7facec0046c0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 -- 172.21.15.77:0/2116286532 shutdown_connections 2024-04-25T01:24:22.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7facd406be60 0x7facd406e320 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7facf015e420 0x7facf00b1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facf00cff30 0x7facf014ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 --2- 172.21.15.77:0/2116286532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf006d240 0x7facf014a2e0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 -- 172.21.15.77:0/2116286532 >> 172.21.15.77:0/2116286532 conn(0x7facf00d1ca0 msgr2=0x7facf00af9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:22.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 -- 172.21.15.77:0/2116286532 shutdown_connections 2024-04-25T01:24:22.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.162+0000 7facf7777640 1 -- 172.21.15.77:0/2116286532 wait complete. 2024-04-25T01:24:22.153 INFO:tasks.ceph:Scrubbing osd.7 2024-04-25T01:24:22.153 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:24:22.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.272+0000 7f3700d10640 1 Processor -- start 2024-04-25T01:24:22.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.272+0000 7f3700d10640 1 -- start start 2024-04-25T01:24:22.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.273+0000 7f3700d10640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f36fc15e4c0 0x7f36fc0d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.273+0000 7f3700d10640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f36fc0d4a10 0x7f36fc0b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.273+0000 7f3700d10640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36fc0d2d40 0x7f36fc0b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.273+0000 7f3700d10640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f36fc0d1e90 con 0x7f36fc0d4a10 2024-04-25T01:24:22.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.273+0000 7f3700d10640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f36fc0b54a0 con 0x7f36fc15e4c0 2024-04-25T01:24:22.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.273+0000 7f3700d10640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f36fc0d2130 con 0x7f36fc0d2d40 2024-04-25T01:24:22.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.273+0000 7f36f9d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f36fc0d4a10 0x7f36fc0b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.273+0000 7f36f9d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f36fc0d4a10 0x7f36fc0b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49922/0 (socket says 172.21.15.77:49922) 2024-04-25T01:24:22.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.273+0000 7f36f9d9b640 1 -- 172.21.15.77:0/4162162492 learned_addr learned my addr 172.21.15.77:0/4162162492 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:22.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.273+0000 7f36fad9d640 1 --2- 172.21.15.77:0/4162162492 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36fc0d2d40 0x7f36fc0b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.274+0000 7f36fa59c640 1 --2- 172.21.15.77:0/4162162492 >> v2:172.21.15.121:3300/0 conn(0x7f36fc15e4c0 0x7f36fc0d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.274+0000 7f36f9d9b640 1 -- 172.21.15.77:0/4162162492 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36fc0d2d40 msgr2=0x7f36fc0b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:22.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.274+0000 7f36f9d9b640 1 --2- 172.21.15.77:0/4162162492 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36fc0d2d40 0x7f36fc0b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.274+0000 7f36f9d9b640 1 -- 172.21.15.77:0/4162162492 >> v2:172.21.15.121:3300/0 conn(0x7f36fc15e4c0 msgr2=0x7f36fc0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:22.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.274+0000 7f36f9d9b640 1 --2- 172.21.15.77:0/4162162492 >> v2:172.21.15.121:3300/0 conn(0x7f36fc15e4c0 0x7f36fc0d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.274+0000 7f36f9d9b640 1 -- 172.21.15.77:0/4162162492 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36fc0b5780 con 0x7f36fc0d4a10 2024-04-25T01:24:22.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.274+0000 7f36fa59c640 1 --2- 172.21.15.77:0/4162162492 >> v2:172.21.15.121:3300/0 conn(0x7f36fc15e4c0 0x7f36fc0d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:22.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.274+0000 7f36f9d9b640 1 --2- 172.21.15.77:0/4162162492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36fc0d4a10 0x7f36fc0b3ee0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f36fc067960 tx=0x7f36f002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c6e2d7b9927688 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.274+0000 7f36f959a640 1 -- 172.21.15.77:0/4162162492 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f0037070 con 0x7f36fc0d4a10 2024-04-25T01:24:22.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.275+0000 7f36f959a640 1 -- 172.21.15.77:0/4162162492 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f0032440 con 0x7f36fc0d4a10 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.275+0000 7f36f959a640 1 -- 172.21.15.77:0/4162162492 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f0032810 con 0x7f36fc0d4a10 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.275+0000 7f3700d10640 1 -- 172.21.15.77:0/4162162492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36fc0d4a10 msgr2=0x7f36fc0b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.275+0000 7f3700d10640 1 --2- 172.21.15.77:0/4162162492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36fc0d4a10 0x7f36fc0b3ee0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f36fc067960 tx=0x7f36f002a140 comp rx=0 tx=0).stop 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.275+0000 7f3700d10640 1 -- 172.21.15.77:0/4162162492 shutdown_connections 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.275+0000 7f3700d10640 1 --2- 172.21.15.77:0/4162162492 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36fc0d2d40 0x7f36fc0b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.275+0000 7f3700d10640 1 --2- 172.21.15.77:0/4162162492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36fc0d4a10 0x7f36fc0b3ee0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.275+0000 7f3700d10640 1 --2- 172.21.15.77:0/4162162492 >> v2:172.21.15.121:3300/0 conn(0x7f36fc15e4c0 0x7f36fc0d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.275+0000 7f3700d10640 1 -- 172.21.15.77:0/4162162492 >> 172.21.15.77:0/4162162492 conn(0x7f36fc0d3ef0 msgr2=0x7f36fc0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.275+0000 7f3700d10640 1 -- 172.21.15.77:0/4162162492 shutdown_connections 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.275+0000 7f3700d10640 1 -- 172.21.15.77:0/4162162492 wait complete. 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.276+0000 7f3700d10640 1 Processor -- start 2024-04-25T01:24:22.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.277+0000 7f3700d10640 1 -- start start 2024-04-25T01:24:22.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.277+0000 7f3700d10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36fc0d2d40 0x7f36fc1f7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.277+0000 7f3700d10640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36fc0d4a10 0x7f36fc0c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.277+0000 7f3700d10640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36fc15e4c0 0x7f36fc157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.277+0000 7f3700d10640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc15ea20 con 0x7f36fc0d2d40 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.277+0000 7f3700d10640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc15e8a0 con 0x7f36fc0d4a10 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.277+0000 7f3700d10640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f36fc0d0900 con 0x7f36fc15e4c0 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.277+0000 7f36fa59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36fc0d2d40 0x7f36fc1f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.277+0000 7f36fa59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36fc0d2d40 0x7f36fc1f7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49928/0 (socket says 172.21.15.77:49928) 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.277+0000 7f36fa59c640 1 -- 172.21.15.77:0/3235195192 learned_addr learned my addr 172.21.15.77:0/3235195192 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36fad9d640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36fc15e4c0 0x7f36fc157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36f9d9b640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36fc0d4a10 0x7f36fc0c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36fa59c640 1 -- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36fc15e4c0 msgr2=0x7f36fc157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36fa59c640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36fc15e4c0 0x7f36fc157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36fa59c640 1 -- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36fc0d4a10 msgr2=0x7f36fc0c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36fa59c640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36fc0d4a10 0x7f36fc0c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36fa59c640 1 -- 172.21.15.77:0/3235195192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36fc1f8380 con 0x7f36fc0d2d40 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36fad9d640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36fc15e4c0 0x7f36fc157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36f9d9b640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36fc0d4a10 0x7f36fc0c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36fa59c640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36fc0d2d40 0x7f36fc1f7790 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f36ec007ef0 tx=0x7f36ec00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36eaffd640 1 -- 172.21.15.77:0/3235195192 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec01a070 con 0x7f36fc0d2d40 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36eaffd640 1 -- 172.21.15.77:0/3235195192 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36ec003b90 con 0x7f36fc0d2d40 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f36eaffd640 1 -- 172.21.15.77:0/3235195192 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec004030 con 0x7f36fc0d2d40 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f3700d10640 1 -- 172.21.15.77:0/3235195192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36fc0ca410 con 0x7f36fc0d2d40 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.278+0000 7f3700d10640 1 -- 172.21.15.77:0/3235195192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36fc0cc1b0 con 0x7f36fc0d2d40 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.279+0000 7f3700d10640 1 -- 172.21.15.77:0/3235195192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f36fc068fb0 con 0x7f36fc0d2d40 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.280+0000 7f36eaffd640 1 -- 172.21.15.77:0/3235195192 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f36ec01d070 con 0x7f36fc0d2d40 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.280+0000 7f36eaffd640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f36dc06be30 0x7f36dc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.280+0000 7f36eaffd640 1 -- 172.21.15.77:0/3235195192 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f36ec08af50 con 0x7f36fc0d2d40 2024-04-25T01:24:22.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.280+0000 7f36f9d9b640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f36dc06be30 0x7f36dc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.280+0000 7f36eaffd640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x7f36dc072410 0x7f36dc074850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.280+0000 7f36eaffd640 1 -- 172.21.15.77:0/3235195192 --> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f36ec011e50 con 0x7f36dc072410 2024-04-25T01:24:22.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.280+0000 7f36eaffd640 1 -- 172.21.15.77:0/3235195192 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f36ec08d5f0 con 0x7f36fc0d2d40 2024-04-25T01:24:22.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.280+0000 7f36fad9d640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x7f36dc072410 0x7f36dc074850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.280+0000 7f36f9d9b640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f36dc06be30 0x7f36dc06e2f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f36f0034000 tx=0x7f36f002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.281+0000 7f36fad9d640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x7f36dc072410 0x7f36dc074850 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.282+0000 7f36eaffd640 1 -- 172.21.15.77:0/3235195192 <== osd.7 v2:172.21.15.121:6800/251765690 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f36ec011e50 con 0x7f36dc072410 2024-04-25T01:24:22.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.294+0000 7f3700d10640 1 -- 172.21.15.77:0/3235195192 --> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f36fc0d45b0 con 0x7f36dc072410 2024-04-25T01:24:22.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f36eaffd640 1 -- 172.21.15.77:0/3235195192 <== osd.7 v2:172.21.15.121:6800/251765690 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f36fc0d45b0 con 0x7f36dc072410 2024-04-25T01:24:22.275 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-25T01:24:22.276 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:24:22.276 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-25T01:24:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f3700d10640 1 -- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x7f36dc072410 msgr2=0x7f36dc074850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f3700d10640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.121:6800/251765690,v1:172.21.15.121:6801/251765690] conn(0x7f36dc072410 0x7f36dc074850 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f3700d10640 1 -- 172.21.15.77:0/3235195192 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f36dc06be30 msgr2=0x7f36dc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f3700d10640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f36dc06be30 0x7f36dc06e2f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f36f0034000 tx=0x7f36f002e000 comp rx=0 tx=0).stop 2024-04-25T01:24:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f3700d10640 1 -- 172.21.15.77:0/3235195192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36fc0d2d40 msgr2=0x7f36fc1f7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f3700d10640 1 --2- 172.21.15.77:0/3235195192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36fc0d2d40 0x7f36fc1f7790 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f36ec007ef0 tx=0x7f36ec00d4a0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f36fad9d640 1 -- 172.21.15.77:0/3235195192 reap_dead start 2024-04-25T01:24:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f3700d10640 1 -- 172.21.15.77:0/3235195192 shutdown_connections 2024-04-25T01:24:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f3700d10640 1 -- 172.21.15.77:0/3235195192 >> 172.21.15.77:0/3235195192 conn(0x7f36fc0d3ef0 msgr2=0x7f36fc15d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f3700d10640 1 -- 172.21.15.77:0/3235195192 shutdown_connections 2024-04-25T01:24:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.295+0000 7f3700d10640 1 -- 172.21.15.77:0/3235195192 wait complete. 2024-04-25T01:24:22.286 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-25T01:24:22.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.403+0000 7f2cb9d42640 1 Processor -- start 2024-04-25T01:24:22.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.403+0000 7f2cb9d42640 1 -- start start 2024-04-25T01:24:22.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.403+0000 7f2cb9d42640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb40d4a10 0x7f2cb40cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.403+0000 7f2cb9d42640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2cb40b3ee0 0x7f2cb40b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.403+0000 7f2cb9d42640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cb40d1e90 con 0x7f2cb415e4c0 2024-04-25T01:24:22.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.403+0000 7f2cb9d42640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f2cb40b54a0 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.403+0000 7f2cb9d42640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2cb40d2130 con 0x7f2cb40d4a10 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.404+0000 7f2cb3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2cb40b3ee0 0x7f2cb40b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.404+0000 7f2cb37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2cb415e4c0 0x7f2cb40d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38422/0 (socket says 172.21.15.77:38422) 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.404+0000 7f2cb37fe640 1 -- 172.21.15.77:0/3286498702 learned_addr learned my addr 172.21.15.77:0/3286498702 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.404+0000 7f2cb3fff640 1 -- 172.21.15.77:0/3286498702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb40d4a10 msgr2=0x7f2cb40cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.404+0000 7f2cb3fff640 1 --2- 172.21.15.77:0/3286498702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb40d4a10 0x7f2cb40cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.404+0000 7f2cb3fff640 1 -- 172.21.15.77:0/3286498702 >> v1:172.21.15.77:6789/0 conn(0x7f2cb415e4c0 legacy=0x7f2cb40d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.404+0000 7f2cb3fff640 1 -- 172.21.15.77:0/3286498702 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb40b5780 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.404+0000 7f2cb27fc640 1 -- 172.21.15.77:0/3286498702 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955291177 0 0) 0x7f2cb40d1e90 con 0x7f2cb415e4c0 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb3fff640 1 --2- 172.21.15.77:0/3286498702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb40b3ee0 0x7f2cb40b4390 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009d90 tx=0x7f2cac02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d2860c8174d1b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb27fc640 1 -- 172.21.15.77:0/3286498702 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac031600 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb27fc640 1 -- 172.21.15.77:0/3286498702 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cac0317a0 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb27fc640 1 -- 172.21.15.77:0/3286498702 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac031b70 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb9d42640 1 -- 172.21.15.77:0/3286498702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb40b3ee0 msgr2=0x7f2cb40b4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb9d42640 1 --2- 172.21.15.77:0/3286498702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb40b3ee0 0x7f2cb40b4390 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009d90 tx=0x7f2cac02a5d0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb9d42640 1 -- 172.21.15.77:0/3286498702 shutdown_connections 2024-04-25T01:24:22.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb9d42640 1 --2- 172.21.15.77:0/3286498702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb40b3ee0 0x7f2cb40b4390 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb9d42640 1 --2- 172.21.15.77:0/3286498702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb40d4a10 0x7f2cb40cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb9d42640 1 -- 172.21.15.77:0/3286498702 >> 172.21.15.77:0/3286498702 conn(0x7f2cb40d3ef0 msgr2=0x7f2cb40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:22.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb9d42640 1 -- 172.21.15.77:0/3286498702 shutdown_connections 2024-04-25T01:24:22.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.405+0000 7f2cb9d42640 1 -- 172.21.15.77:0/3286498702 wait complete. 2024-04-25T01:24:22.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.406+0000 7f2cb9d42640 1 Processor -- start 2024-04-25T01:24:22.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.407+0000 7f2cb9d42640 1 -- start start 2024-04-25T01:24:22.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.407+0000 7f2cb9d42640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb40b3ee0 0x7f2cb414dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.407+0000 7f2cb9d42640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb40d4a10 0x7f2cb414a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.407+0000 7f2cb9d42640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb415e4c0 0x7f2cb40ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.407+0000 7f2cb9d42640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb40b30a0 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.407+0000 7f2cb9d42640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb40b2f20 con 0x7f2cb40d4a10 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.407+0000 7f2cb9d42640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2cb415e8a0 con 0x7f2cb415e4c0 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.407+0000 7f2cb37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb40b3ee0 0x7f2cb414dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.407+0000 7f2cb37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb40b3ee0 0x7f2cb414dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49946/0 (socket says 172.21.15.77:49946) 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.407+0000 7f2cb37fe640 1 -- 172.21.15.77:0/1135124078 learned_addr learned my addr 172.21.15.77:0/1135124078 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.408+0000 7f2cb37fe640 1 -- 172.21.15.77:0/1135124078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb415e4c0 msgr2=0x7f2cb40ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.408+0000 7f2cb37fe640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb415e4c0 0x7f2cb40ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.408+0000 7f2cb37fe640 1 -- 172.21.15.77:0/1135124078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb40d4a10 msgr2=0x7f2cb414a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.408+0000 7f2cb37fe640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb40d4a10 0x7f2cb414a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.408+0000 7f2cb37fe640 1 -- 172.21.15.77:0/1135124078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb40cc730 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.408+0000 7f2cb37fe640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb40b3ee0 0x7f2cb414dc00 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400c460 tx=0x7f2ca400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.408+0000 7f2c9bfff640 1 -- 172.21.15.77:0/1135124078 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4018070 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.408+0000 7f2c9bfff640 1 -- 172.21.15.77:0/1135124078 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca4007500 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.408+0000 7f2cb9d42640 1 -- 172.21.15.77:0/1135124078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb414e140 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.408+0000 7f2c9bfff640 1 -- 172.21.15.77:0/1135124078 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4004a20 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.408+0000 7f2cb9d42640 1 -- 172.21.15.77:0/1135124078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb41f7cd0 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.409+0000 7f2cb9d42640 1 -- 172.21.15.77:0/1135124078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cb4068fb0 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.409+0000 7f2c9bfff640 1 -- 172.21.15.77:0/1135124078 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2ca401b070 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.410+0000 7f2c9bfff640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ca006be30 0x7f2ca006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.410+0000 7f2cb2ffd640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ca006be30 0x7f2ca006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.410+0000 7f2c9bfff640 1 -- 172.21.15.77:0/1135124078 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f2ca4089fc0 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.410+0000 7f2cb2ffd640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ca006be30 0x7f2ca006e2f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40b9b10 tx=0x7f2ca8006bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.414+0000 7f2c9bfff640 1 -- 172.21.15.77:0/1135124078 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2ca4058ac0 con 0x7f2cb40b3ee0 2024-04-25T01:24:22.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.586+0000 7f2cb9d42640 1 -- 172.21.15.77:0/1135124078 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f2cb40cc900 con 0x7f2ca006be30 2024-04-25T01:24:22.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.586+0000 7f2c9bfff640 1 -- 172.21.15.77:0/1135124078 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2cb40cc900 con 0x7f2ca006be30 2024-04-25T01:24:22.567 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 7 to deep-scrub 2024-04-25T01:24:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.588+0000 7f2cb9d42640 1 -- 172.21.15.77:0/1135124078 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ca006be30 msgr2=0x7f2ca006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.588+0000 7f2cb9d42640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ca006be30 0x7f2ca006e2f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40b9b10 tx=0x7f2ca8006bf0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.588+0000 7f2cb9d42640 1 -- 172.21.15.77:0/1135124078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb40b3ee0 msgr2=0x7f2cb414dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.588+0000 7f2cb9d42640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb40b3ee0 0x7f2cb414dc00 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400c460 tx=0x7f2ca400c920 comp rx=0 tx=0).stop 2024-04-25T01:24:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.588+0000 7f2cb9d42640 1 -- 172.21.15.77:0/1135124078 shutdown_connections 2024-04-25T01:24:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.588+0000 7f2cb9d42640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f2ca006be30 0x7f2ca006e2f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.588+0000 7f2cb9d42640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb415e4c0 0x7f2cb40ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.588+0000 7f2cb9d42640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb40d4a10 0x7f2cb414a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.588+0000 7f2cb9d42640 1 --2- 172.21.15.77:0/1135124078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb40b3ee0 0x7f2cb414dc00 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.588+0000 7f2cb9d42640 1 -- 172.21.15.77:0/1135124078 >> 172.21.15.77:0/1135124078 conn(0x7f2cb40d3ef0 msgr2=0x7f2cb40d2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.589+0000 7f2cb9d42640 1 -- 172.21.15.77:0/1135124078 shutdown_connections 2024-04-25T01:24:22.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.589+0000 7f2cb9d42640 1 -- 172.21.15.77:0/1135124078 wait complete. 2024-04-25T01:24:22.580 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:24:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.695+0000 7f87de3a2640 1 Processor -- start 2024-04-25T01:24:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.696+0000 7f87de3a2640 1 -- start start 2024-04-25T01:24:22.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.696+0000 7f87de3a2640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f87d8154ac0 0x7f87d815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.696+0000 7f87de3a2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87d80cf2a0 0x7f87d80711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.696+0000 7f87de3a2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d8071730 0x7f87d806e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.696+0000 7f87de3a2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f87d806d540 con 0x7f87d80cf2a0 2024-04-25T01:24:22.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.696+0000 7f87de3a2640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f87d8154ea0 con 0x7f87d8154ac0 2024-04-25T01:24:22.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.696+0000 7f87de3a2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f87d806d790 con 0x7f87d8071730 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.696+0000 7f87d77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87d80cf2a0 0x7f87d80711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.696+0000 7f87d77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87d80cf2a0 0x7f87d80711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49960/0 (socket says 172.21.15.77:49960) 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.696+0000 7f87d77fe640 1 -- 172.21.15.77:0/145089896 learned_addr learned my addr 172.21.15.77:0/145089896 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.697+0000 7f87d77fe640 1 -- 172.21.15.77:0/145089896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d8071730 msgr2=0x7f87d806e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.697+0000 7f87dc93f640 1 --2- 172.21.15.77:0/145089896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d8071730 0x7f87d806e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.697+0000 7f87d77fe640 1 --2- 172.21.15.77:0/145089896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d8071730 0x7f87d806e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.697+0000 7f87d77fe640 1 -- 172.21.15.77:0/145089896 >> v2:172.21.15.121:3300/0 conn(0x7f87d8154ac0 msgr2=0x7f87d815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.697+0000 7f87d7fff640 1 --2- 172.21.15.77:0/145089896 >> v2:172.21.15.121:3300/0 conn(0x7f87d8154ac0 0x7f87d815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.697+0000 7f87d77fe640 1 --2- 172.21.15.77:0/145089896 >> v2:172.21.15.121:3300/0 conn(0x7f87d8154ac0 0x7f87d815b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.697+0000 7f87d77fe640 1 -- 172.21.15.77:0/145089896 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d806da70 con 0x7f87d80cf2a0 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.697+0000 7f87d77fe640 1 --2- 172.21.15.77:0/145089896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf2a0 0x7f87d80711f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f87d80d0bc0 tx=0x7f87c802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3c19b77e8bc9fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87d6ffd640 1 -- 172.21.15.77:0/145089896 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8037070 con 0x7f87d80cf2a0 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87d6ffd640 1 -- 172.21.15.77:0/145089896 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c8032440 con 0x7f87d80cf2a0 2024-04-25T01:24:22.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87d6ffd640 1 -- 172.21.15.77:0/145089896 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8032810 con 0x7f87d80cf2a0 2024-04-25T01:24:22.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87de3a2640 1 -- 172.21.15.77:0/145089896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf2a0 msgr2=0x7f87d80711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87de3a2640 1 --2- 172.21.15.77:0/145089896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf2a0 0x7f87d80711f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f87d80d0bc0 tx=0x7f87c802a1e0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87de3a2640 1 -- 172.21.15.77:0/145089896 shutdown_connections 2024-04-25T01:24:22.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87de3a2640 1 --2- 172.21.15.77:0/145089896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d8071730 0x7f87d806e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87de3a2640 1 --2- 172.21.15.77:0/145089896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf2a0 0x7f87d80711f0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87de3a2640 1 --2- 172.21.15.77:0/145089896 >> v2:172.21.15.121:3300/0 conn(0x7f87d8154ac0 0x7f87d815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87de3a2640 1 -- 172.21.15.77:0/145089896 >> 172.21.15.77:0/145089896 conn(0x7f87d80b1820 msgr2=0x7f87d80b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87de3a2640 1 -- 172.21.15.77:0/145089896 shutdown_connections 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.698+0000 7f87de3a2640 1 -- 172.21.15.77:0/145089896 wait complete. 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.699+0000 7f87de3a2640 1 Processor -- start 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.699+0000 7f87de3a2640 1 -- start start 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.700+0000 7f87de3a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d8071730 0x7f87d8146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.700+0000 7f87de3a2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d80cf2a0 0x7f87d8144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.700+0000 7f87de3a2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d8154ac0 0x7f87d8142510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.700+0000 7f87de3a2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f87d80cf750 con 0x7f87d8071730 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.700+0000 7f87de3a2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f87d806e030 con 0x7f87d8154ac0 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.700+0000 7f87de3a2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f87d806ea90 con 0x7f87d80cf2a0 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.700+0000 7f87d7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d8071730 0x7f87d8146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.700+0000 7f87d7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d8071730 0x7f87d8146340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49972/0 (socket says 172.21.15.77:49972) 2024-04-25T01:24:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.700+0000 7f87d7fff640 1 -- 172.21.15.77:0/3943418330 learned_addr learned my addr 172.21.15.77:0/3943418330 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.700+0000 7f87dc93f640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d8154ac0 0x7f87d8142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.700+0000 7f87d77fe640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d80cf2a0 0x7f87d8144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87d7fff640 1 -- 172.21.15.77:0/3943418330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d80cf2a0 msgr2=0x7f87d8144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87d7fff640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d80cf2a0 0x7f87d8144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87d7fff640 1 -- 172.21.15.77:0/3943418330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d8154ac0 msgr2=0x7f87d8142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87d7fff640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d8154ac0 0x7f87d8142510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87d7fff640 1 -- 172.21.15.77:0/3943418330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d8144760 con 0x7f87d8071730 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87d77fe640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d80cf2a0 0x7f87d8144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87d7fff640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d8071730 0x7f87d8146340 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00c820 tx=0x7f87cc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87d4ff9640 1 -- 172.21.15.77:0/3943418330 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc015070 con 0x7f87d8071730 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87d4ff9640 1 -- 172.21.15.77:0/3943418330 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87cc004020 con 0x7f87d8071730 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87de3a2640 1 -- 172.21.15.77:0/3943418330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d81469b0 con 0x7f87d8071730 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87d4ff9640 1 -- 172.21.15.77:0/3943418330 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0043f0 con 0x7f87d8071730 2024-04-25T01:24:22.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.701+0000 7f87de3a2640 1 -- 172.21.15.77:0/3943418330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d8146f70 con 0x7f87d8071730 2024-04-25T01:24:22.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.702+0000 7f87de3a2640 1 -- 172.21.15.77:0/3943418330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87d8068fb0 con 0x7f87d8071730 2024-04-25T01:24:22.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.703+0000 7f87d4ff9640 1 -- 172.21.15.77:0/3943418330 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f87cc0077d0 con 0x7f87d8071730 2024-04-25T01:24:22.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.703+0000 7f87d4ff9640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 0x7f87c406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.703+0000 7f87d4ff9640 1 -- 172.21.15.77:0/3943418330 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f87cc08a9a0 con 0x7f87d8071730 2024-04-25T01:24:22.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.703+0000 7f87d77fe640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 0x7f87c406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.703+0000 7f87d77fe640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 0x7f87c406e2f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f87c8034000 tx=0x7f87c802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.707+0000 7f87d4ff9640 1 -- 172.21.15.77:0/3943418330 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f87cc058e50 con 0x7f87d8071730 2024-04-25T01:24:22.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.875+0000 7f87de3a2640 1 -- 172.21.15.77:0/3943418330 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f87d8158390 con 0x7f87c406be30 2024-04-25T01:24:22.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.875+0000 7f87d4ff9640 1 -- 172.21.15.77:0/3943418330 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13490 (secure 0 0 0) 0x7f87d8158390 con 0x7f87c406be30 2024-04-25T01:24:22.856 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:24:22.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.877+0000 7f87de3a2640 1 -- 172.21.15.77:0/3943418330 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 msgr2=0x7f87c406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.877+0000 7f87de3a2640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 0x7f87c406e2f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f87c8034000 tx=0x7f87c802e000 comp rx=0 tx=0).stop 2024-04-25T01:24:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.877+0000 7f87de3a2640 1 -- 172.21.15.77:0/3943418330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d8071730 msgr2=0x7f87d8146340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.877+0000 7f87de3a2640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d8071730 0x7f87d8146340 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00c820 tx=0x7f87cc00cce0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.878+0000 7f87de3a2640 1 -- 172.21.15.77:0/3943418330 shutdown_connections 2024-04-25T01:24:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.878+0000 7f87de3a2640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f87c406be30 0x7f87c406e2f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.878+0000 7f87de3a2640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d8154ac0 0x7f87d8142510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.878+0000 7f87de3a2640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87d80cf2a0 0x7f87d8144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.878+0000 7f87de3a2640 1 --2- 172.21.15.77:0/3943418330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d8071730 0x7f87d8146340 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.878+0000 7f87de3a2640 1 -- 172.21.15.77:0/3943418330 >> 172.21.15.77:0/3943418330 conn(0x7f87d80b1820 msgr2=0x7f87d814c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.878+0000 7f87de3a2640 1 -- 172.21.15.77:0/3943418330 shutdown_connections 2024-04-25T01:24:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:22.878+0000 7f87de3a2640 1 -- 172.21.15.77:0/3943418330 wait complete. 2024-04-25T01:24:22.869 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-04-25T01:24:21.187388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":222808,"kb_used_data":6632,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751912,"statfs":{"total":773094113280,"available":772865957888,"internally_reserved":0,"allocated":6791168,"data_stored":3704616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17778,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-80,"num_read_kb":-94,"num_write":-66,"num_write_kb":-40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001096"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":241,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-25T01:24:15.036794+0000","last_change":"2024-04-25T01:21:34.484756+0000","last_active":"2024-04-25T01:24:15.036794+0000","last_peered":"2024-04-25T01:24:15.036794+0000","last_clean":"2024-04-25T01:24:15.036794+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:24:15.036794+0000","last_undegraded":"2024-04-25T01:24:15.036794+0000","last_fullsized":"2024-04-25T01:24:15.036794+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:21:33.239481+0000","last_clean_scrub_stamp":"2024-04-25T01:21:33.239481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:08.474709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28284,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343556,"statfs":{"total":96636764160,"available":96607801344,"internally_reserved":0,"allocated":1294336,"data_stored":905853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27708,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27708,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27708,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28288,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343552,"statfs":{"total":96636764160,"available":96607797248,"internally_reserved":0,"allocated":1294336,"data_stored":905853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:24:22.872 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T01:21:33.239481+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=1, tm_min=21, tm_sec=33, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=1, tm_min=24, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T01:24:22.872 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T01:24:42.873 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:24:42.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.002+0000 7fc954ccd640 1 Processor -- start 2024-04-25T01:24:42.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.002+0000 7fc954ccd640 1 -- start start 2024-04-25T01:24:42.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc954ccd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc950143d20 0x7fc95015ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:42.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc954ccd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc95006ed60 0x7fc95015f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:42.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc954ccd640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc95015f770 con 0x7fc950143940 2024-04-25T01:24:42.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc954ccd640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc950070e60 con 0x7fc95006ed60 2024-04-25T01:24:42.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc954ccd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc950071140 con 0x7fc950143d20 2024-04-25T01:24:42.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc94e59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc950143940 0x7fc95006e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45602/0 (socket says 172.21.15.77:45602) 2024-04-25T01:24:42.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc94e59c640 1 -- 172.21.15.77:0/2690319330 learned_addr learned my addr 172.21.15.77:0/2690319330 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:42.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc94ed9d640 1 --2- 172.21.15.77:0/2690319330 >> v2:172.21.15.121:3300/0 conn(0x7fc95006ed60 0x7fc95015f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:42.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc94dd9b640 1 --2- 172.21.15.77:0/2690319330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc950143d20 0x7fc95015ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:42.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc94d59a640 1 -- 172.21.15.77:0/2690319330 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3289022940 0 0) 0x7fc95015f770 con 0x7fc950143940 2024-04-25T01:24:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc94d59a640 1 -- 172.21.15.77:0/2690319330 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc938003520 con 0x7fc950143940 2024-04-25T01:24:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc94ed9d640 1 -- 172.21.15.77:0/2690319330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc950143d20 msgr2=0x7fc95015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc94ed9d640 1 --2- 172.21.15.77:0/2690319330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc950143d20 0x7fc95015ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc94d59a640 1 -- 172.21.15.77:0/2690319330 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4213769275 0 0) 0x7fc938003520 con 0x7fc950143940 2024-04-25T01:24:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc94ed9d640 1 -- 172.21.15.77:0/2690319330 >> v1:172.21.15.77:6789/0 conn(0x7fc950143940 legacy=0x7fc95006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.003+0000 7fc94ed9d640 1 -- 172.21.15.77:0/2690319330 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc950071420 con 0x7fc95006ed60 2024-04-25T01:24:42.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.004+0000 7fc94ed9d640 1 --2- 172.21.15.77:0/2690319330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95006ed60 0x7fc95015f230 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc948009dc0 tx=0x7fc94802a610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96c217dc6653216c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:42.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.004+0000 7fc94d59a640 1 -- 172.21.15.77:0/2690319330 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940001670 con 0x7fc95006ed60 2024-04-25T01:24:42.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.004+0000 7fc94d59a640 1 -- 172.21.15.77:0/2690319330 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc948007540 con 0x7fc95006ed60 2024-04-25T01:24:42.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.004+0000 7fc94d59a640 1 -- 172.21.15.77:0/2690319330 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948007910 con 0x7fc95006ed60 2024-04-25T01:24:42.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.005+0000 7fc954ccd640 1 -- 172.21.15.77:0/2690319330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95006ed60 msgr2=0x7fc95015f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:42.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.005+0000 7fc954ccd640 1 --2- 172.21.15.77:0/2690319330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95006ed60 0x7fc95015f230 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc948009dc0 tx=0x7fc94802a610 comp rx=0 tx=0).stop 2024-04-25T01:24:42.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.005+0000 7fc954ccd640 1 -- 172.21.15.77:0/2690319330 shutdown_connections 2024-04-25T01:24:42.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.005+0000 7fc954ccd640 1 --2- 172.21.15.77:0/2690319330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95006ed60 0x7fc95015f230 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:42.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.005+0000 7fc954ccd640 1 --2- 172.21.15.77:0/2690319330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc950143d20 0x7fc95015ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:42.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.005+0000 7fc954ccd640 1 -- 172.21.15.77:0/2690319330 >> 172.21.15.77:0/2690319330 conn(0x7fc9500b1820 msgr2=0x7fc9500b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:42.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.005+0000 7fc954ccd640 1 -- 172.21.15.77:0/2690319330 shutdown_connections 2024-04-25T01:24:42.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.005+0000 7fc954ccd640 1 -- 172.21.15.77:0/2690319330 wait complete. 2024-04-25T01:24:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.006+0000 7fc954ccd640 1 Processor -- start 2024-04-25T01:24:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.006+0000 7fc954ccd640 1 -- start start 2024-04-25T01:24:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.007+0000 7fc954ccd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc95006ed60 0x7fc950157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.007+0000 7fc954ccd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc950143940 0x7fc950154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.007+0000 7fc954ccd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc950143d20 0x7fc9500b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.007+0000 7fc954ccd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc95006d6c0 con 0x7fc950143d20 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.007+0000 7fc954ccd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc95006d540 con 0x7fc950143940 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.007+0000 7fc954ccd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc95006d840 con 0x7fc95006ed60 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.007+0000 7fc94ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc950143d20 0x7fc9500b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.007+0000 7fc94ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc950143d20 0x7fc9500b37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45950/0 (socket says 172.21.15.77:45950) 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.007+0000 7fc94ed9d640 1 -- 172.21.15.77:0/230496614 learned_addr learned my addr 172.21.15.77:0/230496614 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.007+0000 7fc94dd9b640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc950143940 0x7fc950154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.007+0000 7fc94e59c640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc95006ed60 0x7fc950157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.008+0000 7fc94ed9d640 1 -- 172.21.15.77:0/230496614 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc95006ed60 msgr2=0x7fc950157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.008+0000 7fc94ed9d640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc95006ed60 0x7fc950157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.008+0000 7fc94ed9d640 1 -- 172.21.15.77:0/230496614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc950143940 msgr2=0x7fc950154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.008+0000 7fc94ed9d640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc950143940 0x7fc950154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.008+0000 7fc94ed9d640 1 -- 172.21.15.77:0/230496614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9500ce610 con 0x7fc950143d20 2024-04-25T01:24:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.008+0000 7fc94ed9d640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc950143d20 0x7fc9500b37c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc948009ee0 tx=0x7fc948004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.008+0000 7fc93effd640 1 -- 172.21.15.77:0/230496614 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948040070 con 0x7fc950143d20 2024-04-25T01:24:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.008+0000 7fc93effd640 1 -- 172.21.15.77:0/230496614 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc948031bc0 con 0x7fc950143d20 2024-04-25T01:24:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.008+0000 7fc954ccd640 1 -- 172.21.15.77:0/230496614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9500ce8f0 con 0x7fc950143d20 2024-04-25T01:24:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.008+0000 7fc93effd640 1 -- 172.21.15.77:0/230496614 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948009450 con 0x7fc950143d20 2024-04-25T01:24:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.008+0000 7fc954ccd640 1 -- 172.21.15.77:0/230496614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9501543d0 con 0x7fc950143d20 2024-04-25T01:24:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.009+0000 7fc954ccd640 1 -- 172.21.15.77:0/230496614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc950068fb0 con 0x7fc950143d20 2024-04-25T01:24:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.010+0000 7fc93effd640 1 -- 172.21.15.77:0/230496614 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc948039040 con 0x7fc950143d20 2024-04-25T01:24:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.010+0000 7fc93effd640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc93006be30 0x7fc93006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.010+0000 7fc93effd640 1 -- 172.21.15.77:0/230496614 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7fc9480ac390 con 0x7fc950143d20 2024-04-25T01:24:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.010+0000 7fc94e59c640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc93006be30 0x7fc93006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.011+0000 7fc94e59c640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc93006be30 0x7fc93006e2f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc940005880 tx=0x7fc9400057f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:42.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.016+0000 7fc93effd640 1 -- 172.21.15.77:0/230496614 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc94807a960 con 0x7fc950143d20 2024-04-25T01:24:43.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.186+0000 7fc954ccd640 1 -- 172.21.15.77:0/230496614 --> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc95006d150 con 0x7fc93006be30 2024-04-25T01:24:43.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.187+0000 7fc93effd640 1 -- 172.21.15.77:0/230496614 <== mgr.4104 v2:172.21.15.77:6832/436162236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13359 (secure 0 0 0) 0x7fc95006d150 con 0x7fc93006be30 2024-04-25T01:24:43.167 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:43.167 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-25T01:24:43.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 -- 172.21.15.77:0/230496614 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc93006be30 msgr2=0x7fc93006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc93006be30 0x7fc93006e2f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc940005880 tx=0x7fc9400057f0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 -- 172.21.15.77:0/230496614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc950143d20 msgr2=0x7fc9500b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc950143d20 0x7fc9500b37c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc948009ee0 tx=0x7fc948004a40 comp rx=0 tx=0).stop 2024-04-25T01:24:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 -- 172.21.15.77:0/230496614 shutdown_connections 2024-04-25T01:24:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fc93006be30 0x7fc93006e2f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc950143d20 0x7fc9500b37c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc950143940 0x7fc950154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 --2- 172.21.15.77:0/230496614 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc95006ed60 0x7fc950157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 -- 172.21.15.77:0/230496614 >> 172.21.15.77:0/230496614 conn(0x7fc9500b1820 msgr2=0x7fc95015e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 -- 172.21.15.77:0/230496614 shutdown_connections 2024-04-25T01:24:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.189+0000 7fc954ccd640 1 -- 172.21.15.77:0/230496614 wait complete. 2024-04-25T01:24:43.180 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-04-25T01:24:41.191241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":222796,"kb_used_data":6632,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751924,"statfs":{"total":773094113280,"available":772865970176,"internally_reserved":0,"allocated":6791168,"data_stored":3704616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002335"},"pg_stats":[{"pgid":"1.0","version":"58'119","reported_seq":252,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-25T01:24:23.051731+0000","last_change":"2024-04-25T01:24:23.051769+0000","last_active":"2024-04-25T01:24:23.051731+0000","last_peered":"2024-04-25T01:24:23.051731+0000","last_clean":"2024-04-25T01:24:23.051731+0000","last_became_active":"2024-04-25T01:21:34.481080+0000","last_became_peered":"2024-04-25T01:21:34.481080+0000","last_unstale":"2024-04-25T01:24:23.051731+0000","last_undegraded":"2024-04-25T01:24:23.051731+0000","last_fullsized":"2024-04-25T01:24:23.051731+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'119","last_scrub_stamp":"2024-04-25T01:24:23.051638+0000","last_deep_scrub":"58'119","last_deep_scrub_stamp":"2024-04-25T01:24:23.051638+0000","last_clean_scrub_stamp":"2024-04-25T01:24:23.051638+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:09.477504+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1294336,"data_stored":905853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27708,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27708,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27708,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":700416,"data_stored":315485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1294336,"data_stored":905853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:24:43.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T01:24:43.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.297+0000 7f069a410640 1 Processor -- start 2024-04-25T01:24:43.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.298+0000 7f069a410640 1 -- start start 2024-04-25T01:24:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.298+0000 7f069a410640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06940b47d0 0x7f069415b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.298+0000 7f069a410640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06940b37c0 0x7f069415ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.298+0000 7f069a410640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f069415bfa0 con 0x7f0694158a70 2024-04-25T01:24:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.298+0000 7f069a410640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f069415c210 con 0x7f06940b47d0 2024-04-25T01:24:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.298+0000 7f069a410640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f06940b2ce0 con 0x7f06940b37c0 2024-04-25T01:24:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.298+0000 7f06937fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06940b47d0 0x7f069415b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.299+0000 7f06937fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06940b47d0 0x7f069415b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45964/0 (socket says 172.21.15.77:45964) 2024-04-25T01:24:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.299+0000 7f06937fe640 1 -- 172.21.15.77:0/1997169616 learned_addr learned my addr 172.21.15.77:0/1997169616 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.299+0000 7f06989ad640 1 --2- 172.21.15.77:0/1997169616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06940b37c0 0x7f069415ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.299+0000 7f06937fe640 1 -- 172.21.15.77:0/1997169616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06940b37c0 msgr2=0x7f069415ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.299+0000 7f06937fe640 1 --2- 172.21.15.77:0/1997169616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06940b37c0 0x7f069415ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.299+0000 7f06937fe640 1 -- 172.21.15.77:0/1997169616 >> v1:172.21.15.121:6789/0 conn(0x7f0694158a70 legacy=0x7f06940b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.299+0000 7f06937fe640 1 -- 172.21.15.77:0/1997169616 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06940b2fc0 con 0x7f06940b47d0 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.299+0000 7f06989ad640 1 --2- 172.21.15.77:0/1997169616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06940b37c0 0x7f069415ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.299+0000 7f06937fe640 1 --2- 172.21.15.77:0/1997169616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06940b47d0 0x7f069415b520 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f0684009870 tx=0x7f068402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced66d8b840d717a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.300+0000 7f0692ffd640 1 -- 172.21.15.77:0/1997169616 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684037070 con 0x7f06940b47d0 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.300+0000 7f0692ffd640 1 -- 172.21.15.77:0/1997169616 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0684032440 con 0x7f06940b47d0 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.300+0000 7f0692ffd640 1 -- 172.21.15.77:0/1997169616 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684032810 con 0x7f06940b47d0 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.300+0000 7f069a410640 1 -- 172.21.15.77:0/1997169616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06940b47d0 msgr2=0x7f069415b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.300+0000 7f069a410640 1 --2- 172.21.15.77:0/1997169616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06940b47d0 0x7f069415b520 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f0684009870 tx=0x7f068402a210 comp rx=0 tx=0).stop 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.300+0000 7f069a410640 1 -- 172.21.15.77:0/1997169616 shutdown_connections 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.300+0000 7f069a410640 1 --2- 172.21.15.77:0/1997169616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06940b37c0 0x7f069415ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.300+0000 7f069a410640 1 --2- 172.21.15.77:0/1997169616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06940b47d0 0x7f069415b520 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.300+0000 7f069a410640 1 -- 172.21.15.77:0/1997169616 >> 172.21.15.77:0/1997169616 conn(0x7f069406b720 msgr2=0x7f06940b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:43.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.300+0000 7f069a410640 1 -- 172.21.15.77:0/1997169616 shutdown_connections 2024-04-25T01:24:43.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.300+0000 7f069a410640 1 -- 172.21.15.77:0/1997169616 wait complete. 2024-04-25T01:24:43.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.301+0000 7f069a410640 1 Processor -- start 2024-04-25T01:24:43.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f069a410640 1 -- start start 2024-04-25T01:24:43.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f069a410640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06940b37c0 0x7f069413b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f069a410640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06940b47d0 0x7f06941418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f069a410640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0694158a70 0x7f069415ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f069a410640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f069406c9f0 con 0x7f0694158a70 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f069a410640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0694158e50 con 0x7f06940b47d0 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f069a410640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f069406cb70 con 0x7f06940b37c0 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f06989ad640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0694158a70 0x7f069415ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f06989ad640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0694158a70 0x7f069415ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45978/0 (socket says 172.21.15.77:45978) 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f06989ad640 1 -- 172.21.15.77:0/3430253545 learned_addr learned my addr 172.21.15.77:0/3430253545 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f06937fe640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06940b47d0 0x7f06941418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.302+0000 7f0693fff640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06940b37c0 0x7f069413b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.303+0000 7f06989ad640 1 -- 172.21.15.77:0/3430253545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06940b37c0 msgr2=0x7f069413b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.303+0000 7f06989ad640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06940b37c0 0x7f069413b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.303+0000 7f06989ad640 1 -- 172.21.15.77:0/3430253545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06940b47d0 msgr2=0x7f06941418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.303+0000 7f06989ad640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06940b47d0 0x7f06941418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.303+0000 7f06989ad640 1 -- 172.21.15.77:0/3430253545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f069413e980 con 0x7f0694158a70 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.303+0000 7f06989ad640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0694158a70 0x7f069415ed80 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f068c00cf20 tx=0x7f068c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.303+0000 7f0690ff9640 1 -- 172.21.15.77:0/3430253545 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068c01a070 con 0x7f0694158a70 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.303+0000 7f0690ff9640 1 -- 172.21.15.77:0/3430253545 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f068c00d840 con 0x7f0694158a70 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.303+0000 7f0690ff9640 1 -- 172.21.15.77:0/3430253545 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068c00dc30 con 0x7f0694158a70 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.303+0000 7f069a410640 1 -- 172.21.15.77:0/3430253545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06940af7b0 con 0x7f0694158a70 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.303+0000 7f069a410640 1 -- 172.21.15.77:0/3430253545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06940afd30 con 0x7f0694158a70 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.304+0000 7f069a410640 1 -- 172.21.15.77:0/3430253545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0694068fb0 con 0x7f0694158a70 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.305+0000 7f0690ff9640 1 -- 172.21.15.77:0/3430253545 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f068c01d070 con 0x7f0694158a70 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.305+0000 7f0690ff9640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f068006be30 0x7f068006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.305+0000 7f0690ff9640 1 -- 172.21.15.77:0/3430253545 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7f068c08b560 con 0x7f0694158a70 2024-04-25T01:24:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.305+0000 7f0693fff640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f068006be30 0x7f068006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.305+0000 7f0693fff640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f068006be30 0x7f068006e2f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f06941f57b0 tx=0x7f0688007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:43.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.309+0000 7f0690ff9640 1 -- 172.21.15.77:0/3430253545 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f068c059a40 con 0x7f0694158a70 2024-04-25T01:24:43.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.480+0000 7f069a410640 1 -- 172.21.15.77:0/3430253545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f06940b3c70 con 0x7f0694158a70 2024-04-25T01:24:43.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.484+0000 7f0690ff9640 1 -- 172.21.15.77:0/3430253545 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7f068c05dd40 con 0x7f0694158a70 2024-04-25T01:24:43.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.484+0000 7f0690ff9640 1 -- 172.21.15.77:0/3430253545 <== mon.0 v2:172.21.15.77:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f068c062570 con 0x7f0694158a70 2024-04-25T01:24:43.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.486+0000 7f069a410640 1 -- 172.21.15.77:0/3430253545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f068006be30 msgr2=0x7f068006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.486+0000 7f069a410640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f068006be30 0x7f068006e2f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f06941f57b0 tx=0x7f0688007e30 comp rx=0 tx=0).stop 2024-04-25T01:24:43.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.486+0000 7f069a410640 1 -- 172.21.15.77:0/3430253545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0694158a70 msgr2=0x7f069415ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.486+0000 7f069a410640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0694158a70 0x7f069415ed80 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f068c00cf20 tx=0x7f068c00a7d0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.486+0000 7f069a410640 1 -- 172.21.15.77:0/3430253545 shutdown_connections 2024-04-25T01:24:43.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.486+0000 7f069a410640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7f068006be30 0x7f068006e2f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.486+0000 7f069a410640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0694158a70 0x7f069415ed80 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.486+0000 7f069a410640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06940b47d0 0x7f06941418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.486+0000 7f069a410640 1 --2- 172.21.15.77:0/3430253545 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06940b37c0 0x7f069413b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.486+0000 7f069a410640 1 -- 172.21.15.77:0/3430253545 >> 172.21.15.77:0/3430253545 conn(0x7f069406b720 msgr2=0x7f069413f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:43.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.487+0000 7f069a410640 1 -- 172.21.15.77:0/3430253545 shutdown_connections 2024-04-25T01:24:43.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.487+0000 7f069a410640 1 -- 172.21.15.77:0/3430253545 wait complete. 2024-04-25T01:24:43.477 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T01:24:43.478 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:43.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.594+0000 7fb79b36a640 1 Processor -- start 2024-04-25T01:24:43.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.594+0000 7fb79b36a640 1 -- start start 2024-04-25T01:24:43.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.595+0000 7fb79b36a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb79406ed60 0x7fb79415f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.595+0000 7fb79b36a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb794070e60 con 0x7fb794143940 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.595+0000 7fb79b36a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb794071140 con 0x7fb794143d20 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.595+0000 7fb79b36a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb794071420 con 0x7fb79406ed60 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.595+0000 7fb799106640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb794143940 0x7fb79406e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45612/0 (socket says 172.21.15.77:45612) 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.595+0000 7fb799106640 1 -- 172.21.15.77:0/2868730003 learned_addr learned my addr 172.21.15.77:0/2868730003 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.595+0000 7fb799907640 1 --2- 172.21.15.77:0/2868730003 >> v2:172.21.15.121:3300/0 conn(0x7fb79406ed60 0x7fb79415f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.595+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3666538172 0 0) 0x7fb794070e60 con 0x7fb794143940 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.595+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb780003520 con 0x7fb794143940 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3658002534 0 0) 0x7fb780003520 con 0x7fb794143940 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 >> v2:172.21.15.121:3300/0 conn(0x7fb79406ed60 msgr2=0x7fb79415f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 --2- 172.21.15.77:0/2868730003 >> v2:172.21.15.121:3300/0 conn(0x7fb79406ed60 0x7fb79415f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 >> v1:172.21.15.121:6789/0 conn(0x7fb794143d20 legacy=0x7fb79415ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb794071700 con 0x7fb794143940 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 553793685 0 0) 0x7fb7900019f0 con 0x7fb794143940 2024-04-25T01:24:43.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 >> v1:172.21.15.77:6789/0 conn(0x7fb794143940 legacy=0x7fb79406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 --2- 172.21.15.77:0/2868730003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7800039b0 0x7fb780003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 --2- 172.21.15.77:0/2868730003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb780004800 0x7fb780004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 --2- 172.21.15.77:0/2868730003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7800051c0 0x7fb78000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb794071420 con 0x7fb7800039b0 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb78000c060 con 0x7fb7800051c0 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb78000c260 con 0x7fb780004800 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb799106640 1 --2- 172.21.15.77:0/2868730003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7800039b0 0x7fb780003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb799907640 1 --2- 172.21.15.77:0/2868730003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7800051c0 0x7fb78000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb798905640 1 --2- 172.21.15.77:0/2868730003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb780004800 0x7fb780004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb799106640 1 -- 172.21.15.77:0/2868730003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb780004800 msgr2=0x7fb780004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb799106640 1 --2- 172.21.15.77:0/2868730003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb780004800 0x7fb780004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb799106640 1 -- 172.21.15.77:0/2868730003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7800051c0 msgr2=0x7fb78000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:43.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb799106640 1 --2- 172.21.15.77:0/2868730003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7800051c0 0x7fb78000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb799907640 1 -- 172.21.15.77:0/2868730003 reap_dead start 2024-04-25T01:24:43.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.596+0000 7fb799106640 1 -- 172.21.15.77:0/2868730003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb780003520 con 0x7fb7800039b0 2024-04-25T01:24:43.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.597+0000 7fb799106640 1 --2- 172.21.15.77:0/2868730003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7800039b0 0x7fb780003dd0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb79000a840 tx=0x7fb780005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee429c4df540df14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:43.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.597+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79000a460 con 0x7fb7800039b0 2024-04-25T01:24:43.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.597+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fb79002f070 con 0x7fb7800039b0 2024-04-25T01:24:43.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.597+0000 7fb78bfff640 1 -- 172.21.15.77:0/2868730003 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79002bcd0 con 0x7fb7800039b0 2024-04-25T01:24:43.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.597+0000 7fb79b36a640 1 -- 172.21.15.77:0/2868730003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7800039b0 msgr2=0x7fb780003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.597+0000 7fb79b36a640 1 --2- 172.21.15.77:0/2868730003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7800039b0 0x7fb780003dd0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb79000a840 tx=0x7fb780005830 comp rx=0 tx=0).stop 2024-04-25T01:24:43.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.597+0000 7fb79b36a640 1 -- 172.21.15.77:0/2868730003 shutdown_connections 2024-04-25T01:24:43.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.597+0000 7fb79b36a640 1 --2- 172.21.15.77:0/2868730003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7800039b0 0x7fb780003dd0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.597+0000 7fb79b36a640 1 -- 172.21.15.77:0/2868730003 >> 172.21.15.77:0/2868730003 conn(0x7fb7940b1820 msgr2=0x7fb7940b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.598+0000 7fb79b36a640 1 -- 172.21.15.77:0/2868730003 shutdown_connections 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.598+0000 7fb79b36a640 1 -- 172.21.15.77:0/2868730003 wait complete. 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.599+0000 7fb79b36a640 1 Processor -- start 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.599+0000 7fb79b36a640 1 -- start start 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.599+0000 7fb79b36a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7800039b0 0x7fb7940ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.599+0000 7fb79b36a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb794155bb0 0x7fb7940ce110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.599+0000 7fb79b36a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7941f31b0 0x7fb7941f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.599+0000 7fb79b36a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb79415f080 con 0x7fb7941f31b0 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.599+0000 7fb79b36a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb79415ef00 con 0x7fb7800039b0 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.599+0000 7fb79b36a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb79415f200 con 0x7fb794155bb0 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.599+0000 7fb799907640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7941f31b0 0x7fb7941f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb799907640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7941f31b0 0x7fb7941f3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45996/0 (socket says 172.21.15.77:45996) 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb799907640 1 -- 172.21.15.77:0/259786683 learned_addr learned my addr 172.21.15.77:0/259786683 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb799106640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7800039b0 0x7fb7940ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb798905640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb794155bb0 0x7fb7940ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb799907640 1 -- 172.21.15.77:0/259786683 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb794155bb0 msgr2=0x7fb7940ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb799907640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb794155bb0 0x7fb7940ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb799907640 1 -- 172.21.15.77:0/259786683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7800039b0 msgr2=0x7fb7940ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb799907640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7800039b0 0x7fb7940ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb799907640 1 -- 172.21.15.77:0/259786683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7940cd820 con 0x7fb7941f31b0 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb799907640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7941f31b0 0x7fb7941f3630 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb7940cc8c0 tx=0x7fb78c011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb789ffb640 1 -- 172.21.15.77:0/259786683 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c004440 con 0x7fb7941f31b0 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb789ffb640 1 -- 172.21.15.77:0/259786683 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fb78c01b070 con 0x7fb7941f31b0 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb789ffb640 1 -- 172.21.15.77:0/259786683 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c003870 con 0x7fb7941f31b0 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb79b36a640 1 -- 172.21.15.77:0/259786683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb794152d00 con 0x7fb7941f31b0 2024-04-25T01:24:43.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.600+0000 7fb79b36a640 1 -- 172.21.15.77:0/259786683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7941532c0 con 0x7fb7941f31b0 2024-04-25T01:24:43.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.601+0000 7fb79b36a640 1 -- 172.21.15.77:0/259786683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb794068fb0 con 0x7fb7941f31b0 2024-04-25T01:24:43.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.602+0000 7fb789ffb640 1 -- 172.21.15.77:0/259786683 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb78c003a10 con 0x7fb7941f31b0 2024-04-25T01:24:43.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.602+0000 7fb789ffb640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb77806be60 0x7fb77806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:43.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.602+0000 7fb789ffb640 1 -- 172.21.15.77:0/259786683 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5507+0+0 (secure 0 0 0) 0x7fb78c08a900 con 0x7fb7941f31b0 2024-04-25T01:24:43.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.602+0000 7fb799106640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb77806be60 0x7fb77806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:43.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.603+0000 7fb799106640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb77806be60 0x7fb77806e320 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb790033000 tx=0x7fb790007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:43.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.606+0000 7fb789ffb640 1 -- 172.21.15.77:0/259786683 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb78c024020 con 0x7fb7941f31b0 2024-04-25T01:24:43.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.819+0000 7fb79b36a640 1 -- 172.21.15.77:0/259786683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb79406d150 con 0x7fb7941f31b0 2024-04-25T01:24:43.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.819+0000 7fb789ffb640 1 -- 172.21.15.77:0/259786683 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+3875 (secure 0 0 0) 0x7fb78c058c90 con 0x7fb7941f31b0 2024-04-25T01:24:43.800 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-25T01:24:43.800 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5129,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1374960539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1374960539},{"type":"v1","addr":"172.21.15.121:6834","nonce":1374960539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5467,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3910406676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":3910406676},{"type":"v1","addr":"172.21.15.121:6835","nonce":3910406676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6120,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3088617511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3088617511},{"type":"v1","addr":"172.21.15.77:6837","nonce":3088617511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2995463725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2995463725},{"type":"v1","addr":"172.21.15.77:6835","nonce":2995463725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[]} 2024-04-25T01:24:43.800 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 38 2024-04-25T01:24:43.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.821+0000 7fb79b36a640 1 -- 172.21.15.77:0/259786683 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb77806be60 msgr2=0x7fb77806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.821+0000 7fb79b36a640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb77806be60 0x7fb77806e320 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb790033000 tx=0x7fb790007000 comp rx=0 tx=0).stop 2024-04-25T01:24:43.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.821+0000 7fb79b36a640 1 -- 172.21.15.77:0/259786683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7941f31b0 msgr2=0x7fb7941f3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:43.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.821+0000 7fb79b36a640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7941f31b0 0x7fb7941f3630 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb7940cc8c0 tx=0x7fb78c011920 comp rx=0 tx=0).stop 2024-04-25T01:24:43.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.822+0000 7fb79b36a640 1 -- 172.21.15.77:0/259786683 shutdown_connections 2024-04-25T01:24:43.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.822+0000 7fb79b36a640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.77:6832/436162236,v1:172.21.15.77:6833/436162236] conn(0x7fb77806be60 0x7fb77806e320 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.822+0000 7fb79b36a640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7941f31b0 0x7fb7941f3630 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.822+0000 7fb79b36a640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb794155bb0 0x7fb7940ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.822+0000 7fb79b36a640 1 --2- 172.21.15.77:0/259786683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7800039b0 0x7fb7940ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:43.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.822+0000 7fb79b36a640 1 -- 172.21.15.77:0/259786683 >> 172.21.15.77:0/259786683 conn(0x7fb7940b1820 msgr2=0x7fb7940af950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:43.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.822+0000 7fb79b36a640 1 -- 172.21.15.77:0/259786683 shutdown_connections 2024-04-25T01:24:43.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-25T01:24:43.822+0000 7fb79b36a640 1 -- 172.21.15.77:0/259786683 wait complete. 2024-04-25T01:24:43.812 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-25T01:24:43.812 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T01:24:43.812 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T01:24:43.813 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:43.821 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T01:24:43.821 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T01:24:43.821 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:43.871 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T01:24:43.871 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T01:24:43.872 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:43.884 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T01:24:43.884 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T01:24:43.884 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:43.937 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T01:24:43.937 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T01:24:43.937 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T01:24:43.937 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:43.968 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T01:24:43.968 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T01:24:43.968 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.033 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T01:24:44.033 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T01:24:44.033 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.098 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T01:24:44.098 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T01:24:44.098 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-04-25T01:24:44.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-04-25T01:24:44.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-04-25T01:24:44.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-04-25T01:24:44.163 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T01:24:44.164 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-25T01:24:44.164 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.195 INFO:tasks.ceph.osd.4:Stopped 2024-04-25T01:24:44.195 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-25T01:24:44.195 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.271 INFO:tasks.ceph.osd.5:Stopped 2024-04-25T01:24:44.271 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-25T01:24:44.271 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.349 INFO:tasks.ceph.osd.6:Stopped 2024-04-25T01:24:44.349 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-25T01:24:44.350 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.426 INFO:tasks.ceph.osd.7:Stopped 2024-04-25T01:24:44.426 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T01:24:44.427 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-25T01:24:44.427 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.452 INFO:tasks.ceph.mgr.y:Stopped 2024-04-25T01:24:44.453 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T01:24:44.453 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.494 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T01:24:44.494 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T01:24:44.494 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T01:24:44.494 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.505 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T01:24:44.505 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T01:24:44.505 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.546 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T01:24:44.546 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-25T01:24:44.546 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:24:44.599 INFO:tasks.ceph.mon.c:Stopped 2024-04-25T01:24:44.599 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T01:24:44.600 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'force file system read-only' | egrep -v 'bad backtrace' | egrep -v 'MDS in read-only mode' | egrep -v '\(MDS_READ_ONLY\)' | head -n 1 2024-04-25T01:24:44.634 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:24:44.634 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T01:24:44.824 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:24:44.825 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T01:24:44.944 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:24:44.945 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T01:24:44.994 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:24:44.995 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T01:24:45.045 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:24:45.045 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-25T01:24:45.183 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:24:45.184 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-25T01:24:45.251 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:24:45.251 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-25T01:24:45.301 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:24:45.301 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-25T01:24:45.342 INFO:tasks.ceph:Archiving mon data... 2024-04-25T01:24:45.343 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319/data/mon.a.tgz 2024-04-25T01:24:45.343 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-04-25T01:24:45.359 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.aAM66LpChk 2024-04-25T01:24:45.359 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.aAM66LpChk 2024-04-25T01:24:45.503 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.aAM66LpChk 2024-04-25T01:24:45.586 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.aAM66LpChk is 361KB 2024-04-25T01:24:45.683 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.aAM66LpChk 2024-04-25T01:24:45.697 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319/data/mon.b.tgz 2024-04-25T01:24:45.698 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-04-25T01:24:45.715 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.zlk25ilelC 2024-04-25T01:24:45.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.zlk25ilelC 2024-04-25T01:24:45.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.zlk25ilelC 2024-04-25T01:24:45.980 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.zlk25ilelC is 470KB 2024-04-25T01:24:46.077 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.zlk25ilelC 2024-04-25T01:24:46.091 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319/data/mon.c.tgz 2024-04-25T01:24:46.092 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-04-25T01:24:46.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:24:46.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:24:46.145 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.nJt04GQGNa 2024-04-25T01:24:46.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.nJt04GQGNa 2024-04-25T01:24:46.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.nJt04GQGNa 2024-04-25T01:24:46.413 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.nJt04GQGNa is 470KB 2024-04-25T01:24:46.511 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.nJt04GQGNa 2024-04-25T01:24:46.525 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T01:24:46.525 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T01:24:46.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T01:24:46.605 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T01:24:46.642 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T01:24:46.643 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T01:24:46.644 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319/remote/smithi077/crash 2024-04-25T01:24:46.645 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T01:24:46.669 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319/remote/smithi121/crash 2024-04-25T01:24:46.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T01:24:46.705 INFO:tasks.ceph:Shutting down logrotate 2024-04-25T01:24:46.706 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:24:46.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:24:46.807 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T01:24:46.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T01:24:46.853 INFO:tasks.ceph:Compressing logs... 2024-04-25T01:24:46.853 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:24:46.873 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41569.log 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.3.41569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.41569.log.gz 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34706.log 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34575.log 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34706.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34706.log.gz 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34575.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34575.log.gz 2024-04-25T01:24:46.935 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mds.b.log: 92.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34666.log 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr: 93.8%/var/log/ceph/ceph-client.admin.34666.log: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34666.log.gz 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34746.log 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41565.log 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34746.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34746.log.gz 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41165.log 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.3.41565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.41565.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34626.log 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-04-25T01:24:46.936 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.1.41165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.41165.log.gz 2024-04-25T01:24:46.937 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34535.log 2024-04-25T01:24:46.937 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41170.log 2024-04-25T01:24:46.937 INFO:teuthology.orchestra.run.smithi121.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34626.log.gz 2024-04-25T01:24:46.937 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34535.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34535.log.gz 2024-04-25T01:24:46.937 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-25T01:24:46.937 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.1.41170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.41170.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-25T01:24:46.937 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-04-25T01:24:46.937 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mds.d.log: 92.0% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-25T01:24:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34495.log 2024-04-25T01:24:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34786.log 2024-04-25T01:24:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34495.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34495.log.gz 2024-04-25T01:24:46.958 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34786.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34786.log.gz 2024-04-25T01:24:46.964 INFO:teuthology.orchestra.run.smithi121.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-25T01:24:47.036 INFO:teuthology.orchestra.run.smithi121.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-25T01:24:47.047 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51023.log 2024-04-25T01:24:47.047 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43980.log 2024-04-25T01:24:47.047 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51695.log 2024-04-25T01:24:47.047 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44913.log 2024-04-25T01:24:47.047 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51023.log.gz 2024-04-25T01:24:47.047 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51695.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43980.log.gz 2024-04-25T01:24:47.047 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-04-25T01:24:47.047 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51695.log.gz 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46479.log 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54150.log 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49664.log 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.44913.log.gz 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46479.log.gz 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.54150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49199.log 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.54150.log.gz 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40706.log 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49664.log.gz 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42281.log 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.40706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43047.log 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49199.log.gz 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42281.log: gzip -5 88.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40706.log.gz 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr: /var/log/ceph/ceph-client.admin.54454.log 2024-04-25T01:24:47.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43047.log: gzip 89.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.42281.log.gz --verbose 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.48642.log 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54454.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43047.log.gz -5 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.0.50441.log 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48642.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54454.log.gz 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55501.log 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.50441.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48642.log.gz 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53724.log 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50441.log.gz 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49801.log 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50437.log 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55501.log.gz 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41003.log 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53724.log.gz 2024-04-25T01:24:47.049 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.50437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47676.log 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49801.log.gz 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50437.log.gz 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52310.log 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55814.log 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41003.log.gz 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43977.log 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47676.log.gz 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55814.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52310.log.gz 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52412.log 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44601.log 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55814.log.gz 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9%/var/log/ceph/ceph-client.admin.52412.log: -- replaced with /var/log/ceph/ceph-client.admin.43977.log.gz 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46857.log 2024-04-25T01:24:47.050 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42197.log 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52412.log.gz 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46521.log 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44601.log.gz 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46857.log.gz 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53090.log 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46521.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42197.log.gz -5 --verbose 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.46563.log 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53090.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46521.log.gz 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47262.log 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46563.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53090.log.gz 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44661.log 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47262.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46563.log.gz -5 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46605.log 2024-04-25T01:24:47.051 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47262.log.gz 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51914.log 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46605.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44661.log.gz 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48996.log 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46605.log.gz 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52665.log 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51914.log: /var/log/ceph/ceph-client.admin.48996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56958.log 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51914.log.gz 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34383.log 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48996.log.gz 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52665.log.gz 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47550.log 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.34383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52268.log 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56958.log.gz 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42465.log 2024-04-25T01:24:47.052 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34383.log.gz 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47550.log.gz 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52844.log 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52268.log.gz 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46353.log 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46059.log 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42465.log.gz 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46353.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52844.log.gz 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53346.log 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46353.log.gz 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51544.log 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53346.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46059.log.gz 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53850.log 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53346.log.gz 2024-04-25T01:24:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40625.log 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51544.log.gz 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51214.log 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.40625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49326.log 88.7% 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53850.log.gz 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51214.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40625.log.gz 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr: 85.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45975.log 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51214.log.gz 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46899.log 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41361.log 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49326.log.gz 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45975.log.gz 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48558.log 2024-04-25T01:24:47.054 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41361.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46899.log.gz 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48558.log: gzip 87.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41361.log.gz -- 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr: /var/log/ceph/ceph-client.admin.56328.log 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48558.log.gz 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47304.log 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56916.log 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47304.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.56328.log.gz 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52455.log 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47304.log.gz 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55730.log 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47934.log 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56916.log.gz 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52455.log: /var/log/ceph/ceph-client.admin.55730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53682.log 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52455.log.gz 2024-04-25T01:24:47.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47934.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55730.log.gz 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42879.log 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47934.log.gz 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53048.log 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42879.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.53682.log.gz 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42879.log.gz 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49931.log 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph.audit.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53048.log.gz 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50183.log 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50728.log 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50183.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49931.log.gz 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42239.log 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49368.log 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.50183.log.gz 2024-04-25T01:24:47.056 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50728.log.gz 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51501.log 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54146.log 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42239.log.gz 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51501.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.49368.log.gz -5 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53132.log 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51501.log.gz 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.54146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40875.log 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.54146.log.gz 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46143.log 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53132.log.gz 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.40875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55113.log 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46143.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40875.log.gz 2024-04-25T01:24:47.057 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41981.log 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55113.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46143.log.gz 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44871.log 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55113.log.gz 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54945.log 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43978.log 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54945.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44871.log.gz 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49889.log 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43647.log 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54945.log.gz/var/log/ceph/ceph-client.admin.43978.log: 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49889.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43978.log.gz -5 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57040.log 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr:gzip 92.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.41981.log.gz --verbose 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.43641.log 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43647.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49889.log.gz 2024-04-25T01:24:47.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48138.log 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43647.log.gz 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41487.log 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57040.log.gz 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43641.log.gz 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42921.log 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41487.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48138.log.gz 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56748.log 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47760.log 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56748.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42921.log.gz 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49708.log 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41487.log.gz 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56748.log.gz 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47025.log 2024-04-25T01:24:47.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47760.log: /var/log/ceph/ceph-client.admin.49708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56286.log 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47760.log.gz 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47025.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49708.log.gz 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49759.log 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47025.log.gz 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56286.log: /var/log/ceph/ceph-client.admin.49759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47346.log 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56286.log.gz 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.3.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49759.log.gz 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46311.log 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48060.log 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46311.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47346.log.gz 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51956.log 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46311.log.gz 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49494.log 2024-04-25T01:24:47.060 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48060.log: /var/log/ceph/ceph-client.admin.51956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52623.log 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48060.log.gz 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49494.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51956.log.gz 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43311.log 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49494.log.gz 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53388.log 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52623.log.gz 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49241.log 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43311.log.gz 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49536.log 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53388.log: /var/log/ceph/ceph-client.admin.49241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50015.log 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53388.log.gz 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49536.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49241.log.gz 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-25T01:24:47.061 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50015.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49536.log.gz 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51998.log 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50015.log.gz 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42753.log 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.51998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43304.log 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42753.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51998.log.gz 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41193.log 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42155.log 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42753.log.gz 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43304.log: /var/log/ceph/ceph-client.admin.41193.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.0.45396.log -- replaced with /var/log/ceph/ceph-client.admin.43304.log.gz 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41193.log.gz 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55687.log 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42155.log: /var/log/ceph/ceph-client.0.45396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43005.log 2024-04-25T01:24:47.062 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45396.log.gz 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42155.log.gz 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46815.log 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43005.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55687.log.gz 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44703.log 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43005.log.gz 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50980.log 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55375.log 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44703.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46815.log.gz 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55333.log 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44703.log.gz 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55375.log: gzip -5 --verbose 88.3% -- /var/log/ceph/ceph-client.admin.52707.log -- replaced with /var/log/ceph/ceph-client.admin.50980.log.gz 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55333.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55375.log.gz 2024-04-25T01:24:47.063 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48684.log 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55333.log.gz 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52707.log: /var/log/ceph/ceph-client.admin.48684.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52707.log.gz 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48684.log.gz 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49452.log 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47892.log 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55772.log 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47892.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49452.log.gz 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49283.log 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0%/var/log/ceph/ceph-client.admin.55772.log: -- replaced with /var/log/ceph/ceph-client.admin.47892.log.gz 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48726.log 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55772.log.gz 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5/var/log/ceph/ceph-client.admin.49283.log: --verbose -- /var/log/ceph/ceph-client.admin.41277.log 2024-04-25T01:24:47.064 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48726.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49283.log.gz 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48726.log.gz 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41047.log 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54296.log 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41277.log.gz 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43645.log 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41047.log.gz/var/log/ceph/ceph-client.admin.54296.log: 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54296.log.gz 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28354.log 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55459.log 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43645.log: /var/log/ceph/ceph.tmp-client.admin.28354.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28354.log.gz 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43645.log.gz 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56412.log 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56106.log 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55459.log: /var/log/ceph/ceph-client.admin.56412.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55459.log.gz 2024-04-25T01:24:47.065 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42963.log 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56412.log.gz 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56106.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56106.log.gz 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53262.log 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48474.log 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42963.log: /var/log/ceph/ceph-client.admin.53262.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42963.log.gz 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53262.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44307.log 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48474.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48474.log.gz 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56454.log 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48180.log 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44307.log: /var/log/ceph/ceph-client.admin.56454.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44307.log.gz 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.56454.log.gz 2024-04-25T01:24:47.066 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46773.log 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48180.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48180.log.gz 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43310.log 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48912.log 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6%/var/log/ceph/ceph-client.admin.43310.log: -- replaced with /var/log/ceph/ceph-client.admin.46773.log.gz 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55029.log 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43310.log.gz 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54784.log 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48912.log: /var/log/ceph/ceph-client.admin.55029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43312.log 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54784.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55029.log.gz 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48912.log.gz 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54456.log 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43312.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.54784.log.gz 2024-04-25T01:24:47.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51821.log 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43312.log.gz 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44181.log 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1%/var/log/ceph/ceph-client.admin.51821.log: -- replaced with /var/log/ceph/ceph-client.admin.54456.log.gz 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53218.log 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51821.log.gz/var/log/ceph/ceph-client.admin.44181.log: 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51111.log 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44181.log.gz 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50249.log 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53218.log.gz 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43982.log 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51111.log: /var/log/ceph/ceph-client.admin.50249.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50249.log.gz 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.51111.log.gz 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55585.log 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52581.log 2024-04-25T01:24:47.068 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43982.log: /var/log/ceph/ceph-client.admin.55585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48018.log 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52581.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43982.log.gz 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55585.log.gz 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56064.log 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52581.log.gz 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48954.log 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48018.log: /var/log/ceph/ceph-client.admin.56064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43642.log 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48954.log: 88.5% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48018.log.gz 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56064.log.gz 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42837.log 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48954.log.gz 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54455.log 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43642.log: /var/log/ceph/ceph-client.admin.42837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53766.log 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54455.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43642.log.gz 2024-04-25T01:24:47.069 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42837.log.gz 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50099.log 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54455.log.gz 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53766.log: /var/log/ceph/ceph-client.admin.50099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48828.log 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mon.a.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53766.log.gz 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50099.log.gz 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56998.log 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48786.log 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48828.log: /var/log/ceph/ceph-client.admin.56998.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48828.log.gz 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56998.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40917.log 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48786.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48786.log.gz 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42423.log 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51402.log 2024-04-25T01:24:47.070 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.40917.log: /var/log/ceph/ceph-client.admin.42423.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40917.log.gz 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54850.log 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42423.log.gz 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.51402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.51402.log.gz 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55291.log 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54355.log 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54850.log: /var/log/ceph/ceph-client.admin.55291.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54850.log.gz 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44787.log 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55291.log.gz 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54355.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54355.log.gz 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48600.log 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46269.log 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44787.log: /var/log/ceph/ceph-client.admin.48600.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44787.log.gz 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56538.log 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48600.log.gz 2024-04-25T01:24:47.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46269.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46269.log.gz 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47472.log 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55207.log 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56538.log: /var/log/ceph/ceph-client.admin.47472.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56538.log.gz 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47472.log.gz 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51779.log 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43307.log 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55207.log: /var/log/ceph/ceph-client.admin.51779.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55207.log.gz 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51779.log.gz 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45842.log 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43307.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43307.log.gz 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42323.log 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47388.log 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.2.45842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.45842.log.gz 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43089.log 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42323.log.gz 2024-04-25T01:24:47.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47388.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47388.log.gz 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55417.log 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44349.log 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43089.log: /var/log/ceph/ceph-client.admin.55417.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43089.log.gz 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52142.log 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55417.log.gz 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44349.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44349.log.gz 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48870.log 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54696.log 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52142.log: /var/log/ceph/ceph-client.admin.48870.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52142.log.gz 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43131.log 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48870.log.gz 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54696.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54696.log.gz 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56243.log 2024-04-25T01:24:47.073 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40437.log 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43131.log: /var/log/ceph/ceph-client.admin.56243.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43131.log.gz 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56190.log 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56243.log.gz 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.40437.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40437.log.gz 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50602.log 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47430.log 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56190.log: /var/log/ceph/ceph-client.admin.50602.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56190.log.gz 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50602.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49118.log 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47430.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47430.log.gz 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53808.log 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54634.log 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49118.log: /var/log/ceph/ceph-client.admin.53808.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49118.log.gz 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53808.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52040.log 2024-04-25T01:24:47.074 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54634.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54634.log.gz 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55627.log 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53514.log 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52040.log: /var/log/ceph/ceph-client.admin.55627.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52040.log.gz 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48390.log 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55627.log.gz 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53514.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53514.log.gz 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56832.log 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43985.log 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48390.log: /var/log/ceph/ceph-client.admin.56832.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48390.log.gz 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47718.log 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56832.log.gz 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43985.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43985.log.gz 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44559.log 2024-04-25T01:24:47.075 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43635.log 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47718.log: /var/log/ceph/ceph-client.admin.44559.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47718.log.gz 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44559.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41235.log 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43635.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43635.log.gz 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45534.log 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56664.log 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41235.log: /var/log/ceph/ceph-client.admin.45534.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41235.log.gz 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48222.log 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45534.log.gz 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56664.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56664.log.gz 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48348.log 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49973.log 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48222.log: /var/log/ceph/ceph-client.admin.48348.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48222.log.gz 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48348.log.gz 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53598.log 2024-04-25T01:24:47.076 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49973.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49973.log.gz 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42365.log 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53958.log 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53598.log: /var/log/ceph/ceph-client.admin.42365.log: 91.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 86.9% -- replaced with /var/log/ceph/ceph-client.admin.53598.log.gz 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42365.log.gz 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48516.log 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51158.log 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53958.log: gzip/var/log/ceph/ceph-client.admin.48516.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50141.log 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51158.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48516.log.gz 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53958.log.gz 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51158.log.gz 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41107.log 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47850.log 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50141.log.gz 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49410.log 2024-04-25T01:24:47.077 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41107.log.gz 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53892.log 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47850.log.gz 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56148.log 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49410.log: /var/log/ceph/ceph-client.admin.53892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52539.log 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56148.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49410.log.gz 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45838.log 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53892.log.gz 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56148.log.gz 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40529.log 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52539.log.gz 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.2.45838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.45838.log.gz/var/log/ceph/ceph-client.admin.41149.log: 2024-04-25T01:24:47.078 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56706.log 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50770.log 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.40529.log: /var/log/ceph/ceph-client.admin.56706.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40529.log.gz 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56706.log.gz 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55071.log 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46395.log 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50770.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.55071.log: /var/log/ceph/ceph-client.admin.43979.log 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55071.log.gz 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50770.log.gz/var/log/ceph/ceph-client.admin.46395.log: 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56580.log 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46395.log.gz/var/log/ceph/ceph-client.admin.43979.log: 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50644.log 88.7% 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43979.log.gz 2024-04-25T01:24:47.079 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46731.log 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56580.log.gzgzip 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41553.log 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52749.log 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7%/var/log/ceph/ceph-client.admin.46731.log: -- replaced with /var/log/ceph/ceph-client.admin.50644.log.gz 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41403.log 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46731.log.gz 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41553.log.gz 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42507.log 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52749.log.gz 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43649.log 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41403.log: /var/log/ceph/ceph-client.admin.42507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40959.log 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43649.log: 88.6% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42507.log.gz -- replaced with /var/log/ceph/ceph-client.admin.41403.log.gz 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43649.log.gz 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54987.log 2024-04-25T01:24:47.080 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5/var/log/ceph/ceph-client.admin.40959.log: --verbose -- /var/log/ceph/ceph-client.admin.46227.log 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46101.log 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40959.log.gz 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54987.log.gz/var/log/ceph/ceph-client.admin.46227.log: 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53556.log 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40791.log 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46227.log.gz 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53556.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43648.log 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46101.log.gz 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53556.log.gz 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.40791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47067.log 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46017.log 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40791.log.gz 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47067.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43648.log.gz 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52184.log 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47067.log.gz 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47109.log 2024-04-25T01:24:47.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46017.log: /var/log/ceph/ceph-client.admin.52184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43845.log 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46017.log.gz 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5%/var/log/ceph/ceph-client.admin.47109.log: -- replaced with /var/log/ceph/ceph-client.admin.52184.log.gz 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47109.log.gz 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.56874.log 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44829.log 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43845.log.gz 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44433.log 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56874.log.gz 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49620.log 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1%/var/log/ceph/ceph-client.admin.44433.log: -- replaced with /var/log/ceph/ceph-client.admin.44829.log.gz 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47976.log 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44433.log.gz 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43313.log 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49620.log: /var/log/ceph/ceph-client.admin.47976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44265.log 2024-04-25T01:24:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43313.log: 87.9% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49620.log.gz 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47976.log.gz 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46437.log 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43313.log.gz 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5/var/log/ceph/ceph-client.admin.44265.log: --verbose -- /var/log/ceph/ceph-client.admin.42795.log 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46437.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44265.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50057.log 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46437.log.gz 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46185.log 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42795.log: /var/log/ceph/ceph-client.admin.50057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53430.log 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46185.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42795.log.gz 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50057.log.gz 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40833.log 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46185.log.gz 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41319.log 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52924.log 2024-04-25T01:24:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.40833.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53430.log.gz 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40833.log.gz 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41319.log: 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50686.log 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41319.log.gz 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52924.log: -- /var/log/ceph/ceph-client.admin.50847.log 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50686.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52924.log.gzgzip 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53006.log 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50686.log.gz 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51069.log 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50847.log: /var/log/ceph/ceph-client.admin.53006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54742.log 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51069.log: 88.9% 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53006.log.gz 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50847.log.gz 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51069.log.gz 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52082.log 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47151.log 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54742.log.gz 2024-04-25T01:24:47.084 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48264.log 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52082.log.gz 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51406.log 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47151.log: /var/log/ceph/ceph-client.admin.48264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48432.log 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.51406.log: 88.1% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48264.log.gz 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47151.log.gz 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.51406.log.gz 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48306.log 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56790.log 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48432.log.gz 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46647.log 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48306.log.gz 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56496.log 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53640.log 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46647.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56790.log.gz 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56496.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46647.log.gz 2024-04-25T01:24:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50909.log 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56496.log.gz 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mds.a.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53640.log.gz 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46983.log 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49843.log 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50909.log: /var/log/ceph/ceph-client.admin.46983.log: 89.3% 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46983.log.gz 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50909.log.gz 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52352.log 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43305.log 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49843.log.gz 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51653.log 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5%/var/log/ceph/ceph-client.admin.43305.log: -- replaced with /var/log/ceph/ceph-client.admin.52352.log.gz 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49038.log 2024-04-25T01:24:47.086 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43305.log.gz 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56622.log 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2%/var/log/ceph/ceph-client.admin.49038.log: -- replaced with /var/log/ceph/ceph-client.admin.51653.log.gz 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49038.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54457.log 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56622.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56622.log.gz 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42711.log 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49578.log 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54457.log.gz 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55249.log 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49578.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42711.log.gz 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49578.log.gz 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46941.log 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43981.log 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55249.log.gz 2024-04-25T01:24:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53174.log 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43981.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46941.log.gz 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43981.log.gz 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34749.log 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47592.log 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53174.log.gz 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.34749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40748.log 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47592.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34749.log.gz 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47592.log.gz 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42633.log 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.40748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43640.log 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40748.log.gz 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42549.log 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43640.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42633.log.gz 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43640.log.gz 2024-04-25T01:24:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55856.log 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53304.log 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42549.log.gz 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45077.log 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53304.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55856.log.gz 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53304.log.gz 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45401.log 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42591.log 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45077.log.gz 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.45401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45401.log.gz 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34646.log 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.42591.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42591.log.gz 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44745.log 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.34646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51737.log 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34646.log.gz 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52497.log 2024-04-25T01:24:47.089 INFO:teuthology.orchestra.run.smithi077.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44745.log.gz 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51737.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51737.log.gz 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47634.log 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44475.log 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52497.log.gz 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44223.log 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44475.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47634.log.gz 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43309.log 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44223.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44223.log.gz 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46689.log 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53472.log 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46689.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43309.log.gz 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46689.log.gz 2024-04-25T01:24:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44517.log 2024-04-25T01:24:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52226.log 2024-04-25T01:24:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44517.log: /var/log/ceph/ceph-client.admin.53472.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44517.log.gz 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53472.log.gz 2024-04-25T01:24:47.091 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55543.log 2024-04-25T01:24:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56370.log 2024-04-25T01:24:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52226.log: /var/log/ceph/ceph-client.admin.55543.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52226.log.gz 2024-04-25T01:24:47.091 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55543.log.gz 2024-04-25T01:24:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44391.log 2024-04-25T01:24:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56370.log: /var/log/ceph/ceph-client.admin.44391.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56370.log.gz 2024-04-25T01:24:47.091 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44391.log.gz 2024-04-25T01:24:47.092 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-25T01:24:47.092 INFO:teuthology.orchestra.run.smithi121.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-25T01:24:47.224 INFO:teuthology.orchestra.run.smithi077.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T01:24:47.577 INFO:teuthology.orchestra.run.smithi121.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-25T01:24:47.619 INFO:teuthology.orchestra.run.smithi121.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-25T01:24:47.651 INFO:teuthology.orchestra.run.smithi121.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-25T01:24:47.663 INFO:teuthology.orchestra.run.smithi121.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-25T01:24:47.665 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-04-25T01:24:47.665 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m0.758s 2024-04-25T01:24:47.665 INFO:teuthology.orchestra.run.smithi121.stderr:user 0m3.056s 2024-04-25T01:24:47.665 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m0.125s 2024-04-25T01:24:47.666 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T01:24:47.666 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T01:24:47.667 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T01:24:47.710 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T01:24:47.712 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-25T01:24:47.712 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m0.827s 2024-04-25T01:24:47.712 INFO:teuthology.orchestra.run.smithi077.stderr:user 0m3.447s 2024-04-25T01:24:47.712 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m0.309s 2024-04-25T01:24:47.712 INFO:tasks.ceph:Archiving logs... 2024-04-25T01:24:47.713 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319/remote/smithi077/log 2024-04-25T01:24:47.714 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:24:48.497 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319/remote/smithi121/log 2024-04-25T01:24:48.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:24:48.978 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T01:24:48.987 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T01:24:48.987 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:24:48.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:24:49.078 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T01:24:49.087 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T01:24:49.087 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:24:49.090 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:24:49.103 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:24:49.105 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:24:49.108 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:24:49.108 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-04-25T01:24:49.108 INFO:teuthology.orchestra.run.smithi077.stdout:^- hv01.front.sepia.ceph.com 2 6 377 28 +1106us[+1103us] +/- 592ms 2024-04-25T01:24:49.108 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -154us[ -157us] +/- 67ms 2024-04-25T01:24:49.108 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +166us[ +163us] +/- 76ms 2024-04-25T01:24:49.108 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:24:49.109 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:24:49.110 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-04-25T01:24:49.110 INFO:teuthology.orchestra.run.smithi121.stdout:^- hv01.front.sepia.ceph.com 2 6 377 27 +1139us[+1139us] +/- 592ms 2024-04-25T01:24:49.110 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -133us[ -131us] +/- 67ms 2024-04-25T01:24:49.110 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +194us[ +194us] +/- 76ms 2024-04-25T01:24:49.110 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:24:49.111 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T01:24:49.134 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T01:24:49.135 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T01:24:49.162 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:24:49.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-04-25T01:24:49.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-04-25T01:24:49.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-04-25T01:24:49.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-04-25T01:24:49.232 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:24:49.295 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:24:49.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:24:49.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:24:49.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:24:49.350 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T01:24:49.360 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T01:24:49.425 INFO:teuthology.task.internal:Duration was 569.042190 seconds 2024-04-25T01:24:49.425 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T01:24:49.446 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T01:24:49.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:24:49.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:24:49.484 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:24:49.486 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:24:49.838 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T01:24:49.838 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-04-25T01:24:49.839 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:24:49.897 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-04-25T01:24:49.898 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:24:49.928 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T01:24:49.928 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:24:49.941 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:24:49.994 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T01:24:49.994 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:24:50.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:24:50.129 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T01:24:50.145 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T01:24:50.145 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:24:50.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:24:50.192 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T01:24:50.222 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:24:50.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:24:50.244 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-04-25T01:24:50.254 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-04-25T01:24:50.266 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:24:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:24:50.316 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:24:50.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:24:50.331 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T01:24:50.340 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T01:24:50.341 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319/remote/smithi077 2024-04-25T01:24:50.341 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:24:50.389 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672319/remote/smithi121 2024-04-25T01:24:50.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:24:50.421 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T01:24:50.421 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:24:50.425 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:24:50.471 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T01:24:50.483 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T01:24:50.483 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T01:24:50.521 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T01:24:50.521 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:24:50.524 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:24:50.536 INFO:teuthology.orchestra.run.smithi077.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:24 /home/ubuntu/cephtest 2024-04-25T01:24:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:24 /home/ubuntu/cephtest 2024-04-25T01:24:50.539 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T01:24:50.549 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T01:24:50.572 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/auto-repair} duration: 569.0421895980835 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-25T01:24:50.572 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:24:50.660 INFO:teuthology.run:pass